BenchSeg: A Large-Scale Dataset and Benchmark for Multi-View Food Video Segmentation

TL;DR

BenchSeg addresses cross-view generalization and temporal coherence in food segmentation by introducing a large-scale, multi-scene video benchmark with free-motion hemispherical capture. It proposes a three-stage segmentation framework combining per-frame predictions, memory-based propagation, and optional late fusion, enabling temporally coherent masks across diverse viewpoints. A comprehensive evaluation of 20 architectures—including transformer, CNN, and memory-augmented hybrids—reveals that memory-augmented pipelines (e.g., SegMan+XMem2, SeTR-MLA+XMem2) offer the most stable cross-dataset performance, while standard single-frame segmenters degrade under unseen viewpoints; computational trade-offs vary, necessitating deployment-aware choices. The results underscore the importance of temporal continuity for dietary analysis pipelines and point to efficient, scalable memory-based strategies as a promising direction for real-world food-understanding systems, with BenchSeg serving as a rigorous, reusable benchmark for future work.

Abstract

Food image segmentation is a critical task for dietary analysis, enabling accurate estimation of food volume and nutrients. However, current methods suffer from limited multi-view data and poor generalization to new viewpoints. We introduce BenchSeg, a novel multi-view food video segmentation dataset and benchmark. BenchSeg aggregates 55 dish scenes (from Nutrition5k, Vegetables & Fruits, MetaFood3D, and FoodKit) with 25,284 meticulously annotated frames, capturing each dish under free 360° camera motion. We evaluate a diverse set of 20 state-of-the-art segmentation models (e.g., SAM-based, transformer, CNN, and large multimodal) on the existing FoodSeg103 dataset and evaluate them (alone and combined with video-memory modules) on BenchSeg. Quantitative and qualitative results demonstrate that while standard image segmenters degrade sharply under novel viewpoints, memory-augmented methods maintain temporal consistency across frames. Our best model based on a combination of SeTR-MLA+XMem2 outperforms prior work (e.g., improving over FoodMem by ~2.63% mAP), offering new insights into food segmentation and tracking for dietary analysis. We release BenchSeg to foster future research. The project page including the dataset annotations and the food segmentation models can be found at https://amughrabi.github.io/benchseg.

Figures (22)

• Figure 1: Overview of the proposed three-stage food segmentation methodology: (1) keyframe segmentation generates initial masks, (2) temporal propagation transfers them across non-key frames using stored features, and (3) late fusion refines masks by combining propagated and predictions, enabling a reproducible and temporally coherent food-segmentation process. Camera poses shown in green indicate cases where the matching accuracy threshold $mAP \geq 95\%$ is satisfied; poses in red denote those falling below this threshold.
• Figure 2: For illustration, camera locations and orientations were estimated by Colmap for various bounded and unbounded scenes from the V&F, N5k, FKit, and MTF datasets. The first scene on the left is derived from the FoodKit dataset; the second scene is taken from the Vegetables and Fruits dataset; the third scene originates from the MTF dataset; and the final scene is obtained from the N5k dataset.
• Figure 3: Examples of the occlusion issues present in the V&F dataset are highlighted. The figure shows that an obstacle can obscure the food object depending on the camera position.
• Figure 4: Examples of addressing the lighting challenges present in the N5k and V&F datasets. By utilizing both natural and artificial lighting, we emphasize the reflective surfaces. Additionally, we illustrate the low-light conditions that occur in some of these scenarios.
• Figure 5: Examples of tackling the challenges of motion blur found in the Vegetables and Fruits datasets. This problem frequently occurs with cameras in free motion. The samples may appear sharper because the images have been resized to fit the article page.