MetaFood CVPR 2024 Challenge on Physically Informed 3D Food Reconstruction: Methods and Results

TL;DR

The MetaFood CVPR 2024 Challenge targets volume-accurate 3D reconstruction of food items from 2D imagery, addressing the critical need for precise portion estimation in nutrition monitoring. It introduces a dataset of 20 items scanned with a visible checkerboard for real-scale calibration and a two-phase evaluation (volume via Mean Absolute Percentage Error and shape via Chamfer distance after alignment). The top solutions—VolETA, ININ-VIAUN, and FoodRiddle—combine diverse strategies including multi-view reconstruction with neural rendering, diffusion-based single-view methods, and explicit scale estimation to produce calibrated meshes. The study demonstrates promising reconstruction accuracy and lays groundwork for robust, scalable dietary assessment tools, with implications for personal health management and epidemiological nutrition research. improvements in automation, reduced reliance on manual scaling, and broader testing across complex scenes are identified as future directions.

Abstract

The increasing interest in computer vision applications for nutrition and dietary monitoring has led to the development of advanced 3D reconstruction techniques for food items. However, the scarcity of high-quality data and limited collaboration between industry and academia have constrained progress in this field. Building on recent advancements in 3D reconstruction, we host the MetaFood Workshop and its challenge for Physically Informed 3D Food Reconstruction. This challenge focuses on reconstructing volume-accurate 3D models of food items from 2D images, using a visible checkerboard as a size reference. Participants were tasked with reconstructing 3D models for 20 selected food items of varying difficulty levels: easy, medium, and hard. The easy level provides 200 images, the medium level provides 30 images, and the hard level provides only 1 image for reconstruction. In total, 16 teams submitted results in the final testing phase. The solutions developed in this challenge achieved promising results in 3D food reconstruction, with significant potential for improving portion estimation for dietary assessment and nutritional monitoring. More details about this workshop challenge and access to the dataset can be found at https://sites.google.com/view/cvpr-metafood-2024.

Figures (15)

• Figure 1: Sample challenge data for “everything bagel”.
• Figure 2: The team's few-shot approach for estimating food volume in (a) a few shots involves taking ($\mathcal{I}^D$) and food object masks as input. The team starts by selecting keyframes based on the RGB images, removing blurry and highly overlapped images resulting ($I^K$). Then, (b) the team uses PixSfM to estimate camera poses ($C$). Simultaneously, the team segments the reference object using SAM with a segmentation prompt provided by a user. The team then uses the XMem++ method for memory-tracking to produce reference object masks for all frames, using the reference object mask and RGB images. After that, the team applies a binary image segmentation method to RGB images ($I^K$), reference object masks ($M_r$), and food object masks ($M_f$), resulting in RGBA images ($I^R_r$). In contrast, the team transforms the RGBA images and poses to generate meaningful metadata and create modeled data ($D_m$). Next, (c) the team inputs the modeled data into NeuS2 to reconstruct colorful meshes for reference ($R_r$) and food objects ($R_f$). To ensure accuracy, the team uses "Remove Isolated Pieces" with diameter thresholding to clean up the mesh and remove small isolated pieces that do not belong to the reference or food mesh resulting ($\{R^C_r, R^C_f\}$). Finally, the team manually identifies the scaling factor using the reference mesh via MeshLab ($S$). The team fine-tunes the scaling factor using depth information and the food masks and then applies the fine-tuned scaling factor ($S_f$) to the cleaned food mesh to generate a scaled food mesh ($R^F_f$) in meter unit.
• Figure 3: The team manually measures the scaling factor using MeshLab's Measuring tool. The team measures multiple blocks in the reference object mesh; then, the team takes the average of blocks lengths $l_{avg}$.
• Figure 4: The team's one-shot food volume estimation approach. The team begins with a food-segmented image ($I^R_f$), and then uses the One-2-3-45 model to generate a mesh ($R_f$). Next, the team cleans up the isolated pieces that are less than 5% of the ($R_f$) size, resulting in a cleaned food mesh $R^C_f$. Furthermore, the team chooses a scaling factor based on the depth information $S_f$. Finally, the team applies the chosen scaling factor on $R^C_f$ to have a scaled mesh ($R^F_f$) where the team extracts the volume.
• Figure 5: Comparisons to the team's results and ground truth using the challenge dataset. Each scene shows the team's reconstruction (left) and ground truth (right).