Size Matters: Reconstructing Real-Scale 3D Models from Monocular Images for Food Portion Estimation

TL;DR

This work tackles the problem of recovering true real-world scale from monocular images for accurate food-portion estimation. It introduces a two-stage framework: a single-view 3D reconstruction module (One-2-3-45) that generates geometry with arbitrary scale, and a Real-Scale Module that predicts a volume-scale factor using CLIP-based features from the input image and 75 rendered views of the reconstructed model, with the final scale applied via $\sqrt[3]{\hat{v}_{scale}}$. The method achieves about a 30% reduction in mean absolute volume error on MetaFood3D and OmniObject3D, translating volume estimates into caloric values with improved accuracy. By integrating semantic priors from CLIP with geometric renders, the approach enables real-scale 3D modeling for precision nutrition and broader applications in e-commerce and robotics, while maintaining compatibility with multiple reconstruction backbones and rendering setups (e.g., Blender-based renders).

Abstract

The rise of chronic diseases related to diet, such as obesity and diabetes, emphasizes the need for accurate monitoring of food intake. While AI-driven dietary assessment has made strides in recent years, the ill-posed nature of recovering size (portion) information from monocular images for accurate estimation of ``how much did you eat?'' is a pressing challenge. Some 3D reconstruction methods have achieved impressive geometric reconstruction but fail to recover the crucial real-world scale of the reconstructed object, limiting its usage in precision nutrition. In this paper, we bridge the gap between 3D computer vision and digital health by proposing a method that recovers a true-to-scale 3D reconstructed object from a monocular image. Our approach leverages rich visual features extracted from models trained on large-scale datasets to estimate the scale of the reconstructed object. This learned scale enables us to convert single-view 3D reconstructions into true-to-life, physically meaningful models. Extensive experiments and ablation studies on two publicly available datasets show that our method consistently outperforms existing techniques, achieving nearly a 30% reduction in mean absolute volume-estimation error, showcasing its potential to enhance the domain of precision nutrition. Code: https://gitlab.com/viper-purdue/size-matters

Figures (5)

• Figure 1: Restoring real-world scale in single-view 3D reconstruction. Existing single-view reconstruction methods, such as One-2-3-45 liu2023one-2-3-45 and TripoSR tochilkin2024triposr, generate 3D meshes with arbitrary scales leading to unrealistic size interpretation. In contrast, our approach accurately produces 3D objects at a real-world scale, closely matching actual volumes. For visualization, rather than using the actual reconstructed 3D models, we rescale the original 3D model to match the volumes produced by each method. The visualization (left) highlights the disparity (actual size of the reconstruction compared to a human as a size reference) in physical scale, demonstrating the need for accurate real-scale reconstruction, while the scatter plot (right) demonstrates our method’s significantly improved alignment with ground-truth measurements.
• Figure 2: Method Overview. The 3D reconstruction module relies on a single input image and outputs a 3D model using a single-view 3D reconstruction framework. The reconstructed 3D model is used to produce multiple image renders of this model from different viewpoints. Each of these viewpoints, along with the input image, is passed to the Real-Scale Module, where the features are extracted, combined, and put through a network. This network learns the volume scale factor to rescale the reconstructed 3D model to real physical dimensions.
• Figure 3: Real-Scale Module. The features from the input image $x$ and the rendered image $r_i$ are extracted and then concatenated to a $1536\times1$ feature vector. The 3 MLP layers bring down the dimensionality and finally output a single regression value which is the volume scale factor.
• Figure 4: Estimated Volume vs Actual Volume. The predictions of each volume estimation method are compared with the ground-truth volumes, with perfect scores falling on the $y=x$ line. A linear fit line for each method shows the distance of the predictions from the $y=x$ line. The linear fit lines illustrate the deviation of predictions from the ground truth. Our method exhibits the best fit, indicating superior accuracy.
• Figure 5: Prediction distribution across portion sizes. Our method's estimated volumes are compared with ground-truth values for small (0 - 50 mL), medium (50 - 300 mL), and large ($>$300 mL) portions. The close match between the distributions demonstrates the accurate real-scale predictions of our approach across diverse object sizes.