Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion

TL;DR

The paper tackles nutrition estimation from images by introducing the FastFood dataset (84,446 images across 908 fast-food categories with ingredient and nutrition annotations) and a model-agnostic Visual-Ingredient Feature Fusion (VIF$^2$) framework that fuses CLIP-based ingredient embeddings with visual features. It enhances ingredient robustness with synonym augmentation and sampling, and leverages test-time augmented ingredient predictions from large multimodal models with majority voting to reduce hallucinations. Empirical results on FastFood and Nutrition5k across CNN and Transformer backbones show substantial improvements in caloric and macronutrient MAE, e.g., Caloric MAE reduced from $118.04$ to $61.26$ on FastFood with ResNet101+VIF$^2$, and from $103.03$ to $83.75$ on Nutrition5k for Caloric MAE. The work demonstrates the value of integrating ingredient information into nutrition estimation and points toward future use of large multimodal models for enhanced nutrition analysis and personalized diet guidance.

Abstract

Nutrition estimation is an important component of promoting healthy eating and mitigating diet-related health risks. Despite advances in tasks such as food classification and ingredient recognition, progress in nutrition estimation is limited due to the lack of datasets with nutritional annotations. To address this issue, we introduce FastFood, a dataset with 84,446 images across 908 fast food categories, featuring ingredient and nutritional annotations. In addition, we propose a new model-agnostic Visual-Ingredient Feature Fusion (VIF$^2$) method to enhance nutrition estimation by integrating visual and ingredient features. Ingredient robustness is improved through synonym replacement and resampling strategies during training. The ingredient-aware visual feature fusion module combines ingredient features and visual representation to achieve accurate nutritional prediction. During testing, ingredient predictions are refined using large multimodal models by data augmentation and majority voting. Our experiments on both FastFood and Nutrition5k datasets validate the effectiveness of our proposed method built in different backbones (e.g., Resnet, InceptionV3 and ViT), which demonstrates the importance of ingredient information in nutrition estimation. https://huiyanqi.github.io/fastfood-nutrition-estimation/.

Figures (10)

• Figure 1: Distribution of Top20 ingredients in FastFood dataset.
• Figure 2: Overview of our FastFood dataset, highlighting brands, sample images, and annotated ingredients and nutrition.
• Figure 3: Data Expansion for Food Items Using Different Retrieval Methods. The first row shows official food item images, while the second and third rows illustrate data expansion achieved through Image-Based Image Retrieval and Text-Based Image Retrieval, respectively.
• Figure 4: Distribution of calorie in FastFood dataset.
• Figure 5: Distribution of samples in the dataset across different ranges of fat, carbohydrates, and protein content.