NutritionVerse-Direct: Exploring Deep Neural Networks for Multitask Nutrition Prediction from Food Images

TL;DR

NutritionVerse-Direct tackles automated, image-based prediction of multiple nutritional components (calories, mass, protein, fat, carbohydrates) from meals to support elderly dietary tracking. The approach uses a Vision Transformer backbone with a two-layer shared FC block and task-specific heads to perform multitask regression, evaluated on NV-Real; compared against Inception-ResNet and M-AutoE baselines. The ViT-based model achieved a combined $MAE$ of 412.6, a $25.5\%$ improvement over the Inception-ResNet baseline, illustrating the value of transformer-based visual features for nutrition estimation. The work points to practical potential for automated dietary intake estimation and highlights avenues for future improvements, including task-magnitude balancing in loss functions and exploring additional pre-trained weights.

Abstract

Many aging individuals encounter challenges in effectively tracking their dietary intake, exacerbating their susceptibility to nutrition-related health complications. Self-reporting methods are often inaccurate and suffer from substantial bias; however, leveraging intelligent prediction methods can automate and enhance precision in this process. Recent work has explored using computer vision prediction systems to predict nutritional information from food images. Still, these methods are often tailored to specific situations, require other inputs in addition to a food image, or do not provide comprehensive nutritional information. This paper aims to enhance the efficacy of dietary intake estimation by leveraging various neural network architectures to directly predict a meal's nutritional content from its image. Through comprehensive experimentation and evaluation, we present NutritionVerse-Direct, a model utilizing a vision transformer base architecture with three fully connected layers that lead to five regression heads predicting calories (kcal), mass (g), protein (g), fat (g), and carbohydrates (g) present in a meal. NutritionVerse-Direct yields a combined mean average error score on the NutritionVerse-Real dataset of 412.6, an improvement of 25.5% over the Inception-ResNet model, demonstrating its potential for improving dietary intake estimation accuracy.

Figures (3)

• Figure 1: Architecture used for nutrient prediction.
• Figure 2: Food meal example image of a from NV-Real tai2023nutritionversereal.
• Figure 3: Compressed architecture with fewer fully connected layers.