MultiFoodhat: A potential new paradigm for intelligent food quality inspection

TL;DR

This work tackles the generalization gap in food image recognition by proposing a zero-shot, training-free framework that couples a vision–language backbone with multi-agent dialogue. The system employs an Object Perception Token extractor, a multi-turn reasoning loop, and three specialized agents (Food Scientist, Vision Analyst, and Decider) to iteratively refine predictions without labeled fine-tuning. Empirical results on four public datasets show performance competitive with supervised methods and superior to unsupervised baselines, while providing enhanced interpretability through explicit rationales and stepwise reasoning. The approach offers a practical paradigm for intelligent food quality inspection, enabling robust recognition across diverse cuisines and presentation conditions with minimal data requirements.

Abstract

Food image classification plays a vital role in intelligent food quality inspection, dietary assessment, and automated monitoring. However, most existing supervised models rely heavily on large labeled datasets and exhibit limited generalization to unseen food categories. To overcome these challenges, this study introduces MultiFoodChat, a dialogue-driven multi-agent reasoning framework for zero-shot food recognition. The framework integrates vision-language models (VLMs) and large language models (LLMs) to enable collaborative reasoning through multi-round visual-textual dialogues. An Object Perception Token (OPT) captures fine-grained visual attributes, while an Interactive Reasoning Agent (IRA) dynamically interprets contextual cues to refine predictions. This multi-agent design allows flexible and human-like understanding of complex food scenes without additional training or manual annotations. Experiments on multiple public food datasets demonstrate that MultiFoodChat achieves superior recognition accuracy and interpretability compared with existing unsupervised and few-shot methods, highlighting its potential as a new paradigm for intelligent food quality inspection and analysis.

Figures (7)

• Figure 1: Overview of the proposed multi-agent, training-free classification framework. The top row contrasts traditional CNN-based classification with our multi-agent reasoning approach, where multiple specialized agents collaboratively analyze visual input. The bottom row illustrates the pipeline from data collection and pre-processing to multi-agent chat and reasoning, culminating in the final classification output.
• Figure 2: Examples of food image datasets used in this study: (a) Fruit-10, (b) Fruit and Vegetable Disease, (c) Food11, and (d) Food101.
• Figure 3: Overall framework of the proposed MultiFoodChat system. The model employs multi-turn dialogue between domain agents to improve food image classification accuracy, effectively handling fine-grained recognition tasks where visual-only models often fail.
• Figure 4: Multi-agent food classification framework. YOLO detects food items, and a team of agents (food scientist, computer scientist, decision maker) collaboratively reasons to generate the final category and explanation. Note: The prompt shown in the figure is a simplified example.
• Figure 5: Radar charts showing class-wise Precision, Recall, and F1-score of MultiFoodChat across four benchmark food datasets.