Online Class-Incremental Learning For Real-World Food Image Classification

TL;DR

This work tackles real-world food image classification under Online Class Incremental Learning (OCIL), where data streams are evolving, repeated, and imbalanced. It introduces Realistic OCIL (R-OCIL) by a Realistic Data Distribution Module (RDDM) to simulate dietary patterns and a plug-and-play Dynamic Model Update (DMU) to ER-based OCIL methods, selecting the most informative training samples per task. Empirical results on Food-101 and VFN show that DMU consistently improves performance over standard ER baselines, with notable gains in both accuracy and forgetting mitigation, especially under realistic long-term consumption patterns. The approach provides a practical pathway for lifelong learning in real-world dietary monitoring applications, with potential impact on dietary assessment and health monitoring systems.

Abstract

Food image classification is essential for monitoring health and tracking dietary in image-based dietary assessment methods. However, conventional systems often rely on static datasets with fixed classes and uniform distribution. In contrast, real-world food consumption patterns, shaped by cultural, economic, and personal influences, involve dynamic and evolving data. Thus, require the classification system to cope with continuously evolving data. Online Class Incremental Learning (OCIL) addresses the challenge of learning continuously from a single-pass data stream while adapting to the new knowledge and reducing catastrophic forgetting. Experience Replay (ER) based OCIL methods store a small portion of previous data and have shown encouraging performance. However, most existing OCIL works assume that the distribution of encountered data is perfectly balanced, which rarely happens in real-world scenarios. In this work, we explore OCIL for real-world food image classification by first introducing a probabilistic framework to simulate realistic food consumption scenarios. Subsequently, we present an attachable Dynamic Model Update (DMU) module designed for existing ER methods, which enables the selection of relevant images for model training, addressing challenges arising from data repetition and imbalanced sample occurrences inherent in realistic food consumption patterns within the OCIL framework. Our performance evaluation demonstrates significant enhancements compared to established ER methods, showing great potential for lifelong learning in real-world food image classification scenarios. The code of our method is publicly accessible at https://gitlab.com/viper-purdue/OCIL-real-world-food-image-classification

Figures (4)

• Figure 1: R-OCIL for food image classification. The model $h$ learns new food class data sequentially from realistic consumption patterns without accessing previously learned data. These data streams may include repetitions and imbalanced sample distributions. The updated model should classify all food classes it has encountered.
• Figure 2: (a) Conventional ER OCIL strategy, (b) Our proposed realistic setting along with our DMU module. Existing OCIL ER studies icarlMIRGSSASERDVCPCR used (a). Each ER technique employs a different exemplar update and retrieval strategy, as indicated by the corresponding arrows in the figure.
• Figure 3: Overview of proposed method: The left section of the diagram illustrates the Dynamic Model Update (DMU) process, containing two concurrent data inference pathways. The first branch encompasses a virtual model denoted as $VM_1$, which receives the complete batch of data and the corresponding exemplars retrieved from the buffer. Simultaneously, the second branch involves another virtual model named $VM_2$, and receives only the non-repeated data (associated with new class samples) computed based on Equations \ref{['eqn:overview']} and \ref{['eq:non_overlapped_samples']} along with its corresponding exemplars. At any given task $t_i$, the virtual models correspond to the OCIL model trained until task $t_{i-1}$. The strategy for retrieving exemplars and updating the buffer depends on the specific ER method employed in the experiment. The accuracy of these two virtual models is subsequently compared against the accuracy of the initial task, serving as the threshold. Next, the input image sequence and exemplars from the virtual branch that exhibits superior performance are directed into the main OCIL model for training.
• Figure 4: Accuracy of each incremental task in the Gaussian(GAUSS) and Exponential(EXP) distributions for the moderate food consumption category on DVC (with our proposed DMU module) using the Food-101 dataset with 10 tasks.