Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognition

TL;DR

This paper introduces FoodNExTDB, a large, expert-annotated food image database, and a framework to evaluate six Vision-Language Models on dietary assessment tasks. It proposes Expert-Weighted Recall to account for annotator variability and demonstrates that closed-source models outperform open-source ones, especially in single-product recognition, while fine-grained cooking styles remain challenging. The study highlights significant gaps in current VLM capabilities for automatic dietary assessment and suggests directions for future work in context-aware multimodal nutrition analysis and dataset refinement.

Abstract

Automatic dietary assessment based on food images remains a challenge, requiring precise food detection, segmentation, and classification. Vision-Language Models (VLMs) offer new possibilities by integrating visual and textual reasoning. In this study, we evaluate six state-of-the-art VLMs (ChatGPT, Gemini, Claude, Moondream, DeepSeek, and LLaVA), analyzing their capabilities in food recognition at different levels. For the experimental framework, we introduce the FoodNExTDB, a unique food image database that contains 9,263 expert-labeled images across 10 categories (e.g., "protein source"), 62 subcategories (e.g., "poultry"), and 9 cooking styles (e.g., "grilled"). In total, FoodNExTDB includes 50k nutritional labels generated by seven experts who manually annotated all images in the database. Also, we propose a novel evaluation metric, Expert-Weighted Recall (EWR), that accounts for the inter-annotator variability. Results show that closed-source models outperform open-source ones, achieving over 90% EWR in recognizing food products in images containing a single product. Despite their potential, current VLMs face challenges in fine-grained food recognition, particularly in distinguishing subtle differences in cooking styles and visually similar food items, which limits their reliability for automatic dietary assessment. The FoodNExTDB database is publicly available at https://github.com/AI4Food/FoodNExtDB.

Figures (4)

• Figure 1: Overview of the proposed framework. (A) The FoodNExTDB consists of 9,263 food images labeled by nutrition experts across 10 food categories, 62 subcategories, and 9 cooking styles, with approximately 50,000 assigned labels. (B) The experimental setup, where six Vision-Language Models (VLMs) process food images using a structured prompt and generate predictions. These predictions are then compared against expert annotations to assess model performance in food product detection and recognition.
• Figure 2: Illustration of the proposed Expert-Weighted Recall (EWR) computation for a food image $i$ (left). This graphical example compares annotations (e.g. label $l_1=\textit{Yogurt}$ for product $p_1$) from three nutrition experts (middle) with the predictions made by a VLM (right). The proposed EWR metric reflects how well the VLM aligns with expert consensus while accounting for annotation variability.
• Figure 3: Examples of VLMs predictions compared to nutritionist's annotations. (A) A multi-component dish where some experts identify individual ingredients ("legumes", "meat", "vegetables"), while others classify it as "Spanish stew (cocido") (B) A whole grain bread misclassified by Gemini as "croquettes" due to shape similarity. (C) An orange juice image where some VLMs such as DeepSeek incorrectly identifies a sandwich from a background image on the paper tray liner.
• Figure 4: Radar charts illustrating VLM performance in fine-grained food recognition. We include some examples of all available classes considered in our proposed FoodNExTDB, i.e., 10 categories, 62 subcategories, and 9 cooking styles. (This figure is best viewed in color.)