LLMs-based Augmentation for Domain Adaptation in Long-tailed Food Datasets

TL;DR

We address domain shift, long-tailed distributions, and fine-grained distinctions in real-world food recognition by using large language models to generate title and ingredient texts for food images. These texts are embedded alongside visual features in a shared space via a ResNet-50 image encoder and a hierarchical Transformer text encoder, trained with a cross-modal alignment loss and a calibration-based loss to handle imbalance. The approach yields state-of-the-art performance on two imbalanced, cross-domain Food datasets, with notable gains for tail and fine-grained classes, and demonstrates robustness to different LLMs for text generation. The method offers a simple, scalable cross-modal augmentation that leverages semantic cues from generated text to bridge domain gaps and improve recognition in real-world dietary assessment scenarios.

Abstract

Training a model for food recognition is challenging because the training samples, which are typically crawled from the Internet, are visually different from the pictures captured by users in the free-living environment. In addition to this domain-shift problem, the real-world food datasets tend to be long-tailed distributed and some dishes of different categories exhibit subtle variations that are difficult to distinguish visually. In this paper, we present a framework empowered with large language models (LLMs) to address these challenges in food recognition. We first leverage LLMs to parse food images to generate food titles and ingredients. Then, we project the generated texts and food images from different domains to a shared embedding space to maximize the pair similarities. Finally, we take the aligned features of both modalities for recognition. With this simple framework, we show that our proposed approach can outperform the existing approaches tailored for long-tailed data distribution, domain adaptation, and fine-grained classification, respectively, on two food datasets.

Figures (5)

• Figure 1: The FoodAI-HPB dataset highlights the challenges in real-world food datasets. (a) Domain shifts: a noticeable disparity exists between the source and target domains concerning factors such as background, lighting, food shapes, and serving styles. (b) Distribution imbalance: both domains exhibit an imbalanced nature with distinct categorical distributions. For example, some classes (e.g., chicken pongteh and makhani dal shown in the plot) suffer from being under-represented in both the source and target domains. (c) Fine-grained categorization: food classes that are highly similar but differ in their ingredient compositions. The chicken rice in the source domain is visually similar to the boiled kampung chicken in the target domain. Similarly, the curry puff and tuna puff resemble each other despite belonging to different categories.
• Figure 2: An overview of our proposed framework. The food title and ingredients are generated using LLMs and are encoded by the hierarchical transformers as the embedding $e_{T}$. An image encoder takes the image as input and outputs the visual embedding $e_I$. We align the textual and visual embeddings by enforcing an alignment loss on them and the final embedding is obtained by adding the two modalities. We employ a variant of cross-entropy loss by calibrating the exponentiated logits with their class frequencies to compute the recognition penalties.
• Figure 3: Visualization of (a) domain alignment, (b) imbalance learning, and (c) fine-grained categories using t-SNE van2008visualizing on FoodAI-HPB. Different colors indicate different classes. The star denotes features from the source domain while the triangle represents features from the target domain.
• Figure 4: One example providing insights on text augmentation. The correct food class of the input image is "drunken prawn". By using only visual features (a), the predicted category is "steamed crab". By augmenting with LLM-generated title (b), a better prediction "stir fried garlic chives with prawn" is obtained. If both the title and ingredients are augmented (c), the correct food class is predicted.
• Figure 5: Top-1 accuracies of (a) the baseline method, ERM, (b) our proposed approach for different class groups of source and target domains on FoodAI-HPB.