Dietary Intake Estimation via Continuous 3D Reconstruction of Food

TL;DR

The paper addresses dietary intake estimation by reconstructing 3D food geometry from monocular video to capture volume changes during consumption. It builds on the HOLD pipeline, integrating COLMAP-based pose estimation, HAMER/MANO hand meshes, and SDF refinement to model dynamic hand–food interactions. It evaluates on toy and real food items against water-displacement ground truth, reporting reasonable volumetric accuracy across items and highlighting topology-change detection as a key contribution. The work advances automated dietary monitoring by providing geometry-aware intake estimates without wearables, while noting challenges in occlusions, reflectivity, and the need for automatic state separation for fully automated analysis.

Abstract

Monitoring dietary habits is crucial for preventing health risks associated with overeating and undereating, including obesity, diabetes, and cardiovascular diseases. Traditional methods for tracking food intake rely on self-reported data before or after the eating, which are prone to inaccuracies. This study proposes an approach to accurately monitor ingest behaviours by leveraging 3D food models constructed from monocular 2D video. Using COLMAP and pose estimation algorithms, we generate detailed 3D representations of food, allowing us to observe changes in food volume as it is consumed. Experiments with toy models and real food items demonstrate the approach's potential. Meanwhile, we have proposed a new methodology for automated state recognition challenges to accurately detect state changes and maintain model fidelity. The 3D reconstruction approach shows promise in capturing comprehensive dietary behaviour insights, ultimately contributing to the development of automated and accurate dietary monitoring tools.

Figures (10)

• Figure 1: A high-level diagram of the 3D food construction process. The purple blocks represent the steps of the base HOLD pipeline fan2024hold, while the blue blocks indicate the new components added for sequence separation. The process begins with splitting the captured video into individual frames and segmenting the frames. The object masks are passed into COLMAP to construct the object point cloud, while hand masks are processed by HAMER and refined with MANO to generate the hand meshes. During the point cloud construction, frames are converted to normal maps to detect changes in the original topology to split the original dataset into individual sequences. The HOLD optimizer takes the separated hand meshes and object point cloud as inputs to optimize the surface and scale based on their interaction. At last, the optimized meshes and point cloud is then inputed to train an SDF model, which generates the refined object's mesh.
• Figure 2: Top row: The actual toy triceratops experimented with | Bottom row: The 3D meshes of triceratops in with varied structures | Left: Before detachment of the leg | Right: After detachment of the leg
• Figure 3: The rendered view of triceratops with hands in different topology. Purple colour mesh is the object and the hands' meshes are in white colour. Left: Before detachment | Right: After detachment
• Figure 4: Failure case in the rendered view of triceratops, the red circle indicate the failing part
• Figure 5: The 3D meshes of sandwich in with varied structures | Left: Before bite, the red circle highlight the missing part of the topology from the original sandwich object | Right: After bite