A Computer Vision Challenge

Some people type into some food log a list of items they ate and get an estimate daily calorie intake. Some, find it impractical. My daughter says that she would like her activity tracker look at her plate of food at each meal and update its log! I thought it was crazy to ask for that when I heard it two years ago. I am wiser now, and think this can be achieved by a number of steps of innovation. As a first step, I announce this challenge, named Preethi’s Challenge in honor of the original thinker!

1.   The challenge is to recognize the items and quantities of food from their photographs. I visualize that you would use a tablet to take the picture and run the app on the same device. However, you are free to do it any other way, like using a webcam or a smart phone for picture taking and a laptop to run the app.

2.   If you have bought ready-to-eat food and its container has a bar code to identify it, the app is welcome to scan it and use the info.

3.   The app would use pictures with training information entered into “learning software” that will use machine learning techniques to recognize the food item and estimate its quantity and calorie content.

4.   You will train the app, giving it photographs of named objects annotating them with weight and information like calories/hundred-grams. Don’t worry about the training effort. Training one app would serve millions of users who are going to use copies of that app!

5.   You can use a standard plate of known diameter.

6.   You could also have only one item of food on the plate at a given time (at least at the start of the project).

7.   A good solution is to have the photographs taken by looking down, say at 45 degrees below the horizontal. This would give information about the item of food in X, Y and Z dimensions. The plate’s outline provides information on the scale. If the app finds it difficult to estimate calories from that photograph, it could ask you for a second photograph taken from another angle.

8.   The app should calculate the calorie content to some reasonable accuracy, say 10%.

9.   You can allow the app to ask for additional information when it finds it necessary, for instance, when it sees a plate with vegetables. It could ask, say, if it is boiled potatoes.

10.If you want to cheat, you could use a small plate that sits on a kitchen weighing device showing the total weight in digital form. Your app should recognize the numbers.  It could of course use stored information on the weight of the plate and deduct that.

11.You can make a second version of the app, which should take in more complex photographs, say with multiple food items, to make the app easier to use.

One last word: Use every trick you can think of to make this an easy-to-use app. Technology alone may not provide the best solution!