Ostrakon-VL: Towards Domain-Expert MLLM for Food-Service and Retail Stores

TL;DR

This work targets practical FSRS deployment of multimodal LLMs by addressing three gaps: domain-specific capability misalignment, noisy real-world data, and evaluation misalignment. It introduces Ostrakon-VL, an FSRS-tuned MLLM built on Qwen3-VL-8B, a public FSRS benchmark ShopBench spanning single-image, multi-image, and video inputs, and QUAD, a four-stage data-curation pipeline that compresses 69.25M raw samples to 3.40M high-signal instances while improving downstream performance. The training pipeline combines caption bootstrapping, offline curriculum learning, and mixed preference optimization to achieve strong FSRS perception and reasoning with superior parameter efficiency (e.g., Ostrakon-VL achieves 60.1 on ShopBench, surpassing larger baselines). The results demonstrate robust domain-specific capabilities without catastrophic forgetting of general multimodal skills, and the authors provide reproducible access to Ostrakon-VL and ShopBench, facilitating future FSRS research and deployment. Overall, the work offers a practical, reproducible pathway for domain-expert MLLMs in food-service and retail contexts, emphasizing high-quality data curation and targeted evaluation to drive reliable end-to-end perception-to-decision systems.

Abstract

Multimodal Large Language Models (MLLMs) have recently achieved substantial progress in general-purpose perception and reasoning. Nevertheless, their deployment in Food-Service and Retail Stores (FSRS) scenarios encounters two major obstacles: (i) real-world FSRS data, collected from heterogeneous acquisition devices, are highly noisy and lack auditable, closed-loop data curation, which impedes the construction of high-quality, controllable, and reproducible training corpora; and (ii) existing evaluation protocols do not offer a unified, fine-grained and standardized benchmark spanning single-image, multi-image, and video inputs, making it challenging to objectively gauge model robustness. To address these challenges, we first develop Ostrakon-VL, an FSRS-oriented MLLM based on Qwen3-VL-8B. Second, we introduce ShopBench, the first public benchmark for FSRS. Third, we propose QUAD (Quality-aware Unbiased Automated Data-curation), a multi-stage multimodal instruction data curation pipeline. Leveraging a multi-stage training strategy, Ostrakon-VL achieves an average score of 60.1 on ShopBench, establishing a new state of the art among open-source MLLMs with comparable parameter scales and diverse architectures. Notably, it surpasses the substantially larger Qwen3-VL-235B-A22B (59.4) by +0.7, and exceeds the same-scale Qwen3-VL-8B (55.3) by +4.8, demonstrating significantly improved parameter efficiency. These results indicate that Ostrakon-VL delivers more robust and reliable FSRS-centric perception and decision-making capabilities. To facilitate reproducible research, we will publicly release Ostrakon-VL and the ShopBench benchmark.

Figures (15)

• Figure 1: Overall framework of Ostrakon-VL. The framework consists of two core components: (Top) QUAD, a high-quality data curation pipeline that distills raw data into a high-signal corpus through quality filtering, foundation model referenced filtering multimodal semantic deduplication, and capability coverage redistribution; (Bottom) Training Strategy, a multi-stage process starting from domain knowledge injection via caption bootstrapping, followed by offline curriculum learning for progressive adaptation, and concluding with Mixed Preference Optimization to ensure output stability and robustness.
• Figure 2: ShopBench taxonomy (L1--L4). We unify diverse tasks labels into a canonical L4 label space for consistent evaluation. For visual clarity, L4 categories with limited sample counts are omitted from the figure.
• Figure 3: Counts and Proportions of Samples across Input Subtasks and Output Formats
• Figure 4: Distribution of different Benchmarks. OC-8(Opencompass prevailing used eight benchmarks) and ShopBench(ShopFront, ShopInterior and Kitchen)
• Figure 5: Data distribution across three subsets before and after the redistribution stage.