MetaFood3D: 3D Food Dataset with Nutrition Values
Yuhao Chen, Jiangpeng He, Gautham Vinod, Siddeshwar Raghavan, Chris Czarnecki, Jinge Ma, Talha Ibn Mahmud, Bruce Coburn, Dayou Mao, Saeejith Nair, Pengcheng Xi, Alexander Wong, Edward Delp, Fengqing Zhu
TL;DR
MetaFood3D tackles the lack of nutrition-annotated 3D food datasets by introducing a real-scan collection of 743 textured 3D food objects across 131 categories, linked to FNDDS nutrition codes and accompanied by weight, RGB-D video, depth maps, and segmentation masks. The dataset supports rich modalities and fine-grained food-item mappings (220 items with unique codes), enabling nutrition-aware perception, 3D reconstruction, and synthetic data generation; notably, nutrient content per item is computed as $n_i = \frac{w_i}{100} \cdot d_i$ with $d_i=[e_i,p_i,c_i,f_i]$. Across four downstream tasks, MetaFood3D demonstrates that 3D information improves food portion estimation and enables realistic eating-scene synthesis via GET3D and Blender, while revealing challenges in novel-view synthesis on video data and robustness to intra-class shape diversity. The work establishes a versatile benchmark for dietary assessment and 3D food research, supporting sim-to-real data generation and nutrition-aware DL models for practical health monitoring.
Abstract
Food computing is both important and challenging in computer vision (CV). It significantly contributes to the development of CV algorithms due to its frequent presence in datasets across various applications, ranging from classification and instance segmentation to 3D reconstruction. The polymorphic shapes and textures of food, coupled with high variation in forms and vast multimodal information, including language descriptions and nutritional data, make food computing a complex and demanding task for modern CV algorithms. 3D food modeling is a new frontier for addressing food related problems, due to its inherent capability to deal with random camera views and its straightforward representation for calculating food portion size. However, the primary hurdle in the development of algorithms for food object analysis is the lack of nutrition values in existing 3D datasets. Moreover, in the broader field of 3D research, there is a critical need for domain-specific test datasets. To bridge the gap between general 3D vision and food computing research, we introduce MetaFood3D. This dataset consists of 743 meticulously scanned and labeled 3D food objects across 131 categories, featuring detailed nutrition information, weight, and food codes linked to a comprehensive nutrition database. Our MetaFood3D dataset emphasizes intra-class diversity and includes rich modalities such as textured mesh files, RGB-D videos, and segmentation masks. Experimental results demonstrate our dataset's strong capabilities in enhancing food portion estimation algorithms, highlight the gap between video captures and 3D scanned data, and showcase the strengths of MetaFood3D in generating synthetic eating occasion data and 3D food objects.