FoodTrack: Estimating Handheld Food Portions with Egocentric Video
Ervin Wang, Yuhao Chen
TL;DR
FoodTrack addresses the challenge of estimating the volume of hand-held food from egocentric video, improving over gesture-based intake estimates by directly reconstructing 3D food volume. It integrates super-resolution, segmentation (Grounded SAM, Cutie), depth estimation (ChronoDepth), and 3D mesh generation with BundleSDF, followed by a pose-aware scaling step using $D$ and $f_x$ to compute volume as $V^{(RGB)} = \mathrm{mesh.volume} \cdot (D/f_x)^3$. The approach reduces reliance on fixed bite sizes and performs absolute-volume estimation under occlusions, achieving approximately $7.01\%$ absolute error on a sandwich example—better than a prior method's $16.40\%$ under stricter data collection. The work lays a foundation for real-world dietary monitoring with potential health applications, while future work targets improved 3D reconstruction and bite-level volume accuracy.
Abstract
Accurately tracking food consumption is crucial for nutrition and health monitoring. Traditional approaches typically require specific camera angles, non-occluded images, or rely on gesture recognition to estimate intake, making assumptions about bite size rather than directly measuring food volume. We propose the FoodTrack framework for tracking and measuring the volume of hand-held food items using egocentric video which is robust to hand occlusions and flexible with varying camera and object poses. FoodTrack estimates food volume directly, without relying on intake gestures or fixed assumptions about bite size, offering a more accurate and adaptable solution for tracking food consumption. We achieve absolute percentage loss of approximately 7.01% on a handheld food object, improving upon a previous approach that achieved a 16.40% mean absolute percentage error in its best case, under less flexible conditions.