VIOLA: Towards Video In-Context Learning with Minimal Annotations

TL;DR

VIOLA tackles the practical challenge of adapting video-capable Multimodal LLMs under sparse supervision. It introduces a unified SSAL-inspired pipeline with density-uncertainty-weighted selective annotation, in-context pseudo-annotation, and confidence-aware retrieval and prompting to form a reliable hybrid demonstration pool for inference. Across nine diverse datasets and four MLLMs, VIOLA yields substantial gains over zero-shot and existing baselines, especially at low annotation budgets, highlighting its robustness and practical impact for real-world deployment. The approach provides a concrete, low-cost strategy for domain adaptation of video-language models in industrial, medical, and surveillance contexts.

Abstract

Generalizing Multimodal Large Language Models (MLLMs) to novel video domains is essential for real-world deployment but remains challenging due to the scarcity of labeled data. While In-Context Learning (ICL) offers a training-free adaptation path, standard methods rely on large annotated pools, which are often impractical in specialized environments like industrial or surgical settings since they require the experts' annotations. To bridge this gap, we introduce VIOLA (Video In-cOntext Learning with minimal Annotation), a label-efficient framework that synergizes minimal expert supervision with abundant unlabeled data. First, to maximize the efficiency of a strict annotation budget, we propose density-uncertainty-weighted sampling. Unlike standard diversity or uncertainty strategies that risk selecting visual outliers, our method leverages density estimation to identify samples that are simultaneously diverse, representative, and informative. Second, to utilize the remaining unlabeled data without noise propagation, we construct a hybrid pool and introduce confidence-aware retrieval and confidence-aware prompting. These mechanisms explicitly model label reliability, retrieving demonstrations based on a composite score of similarity and confidence while enabling the MLLM to adaptively distinguish between verified ground truths and noisy pseudo-labels. Extensive experiments across nine diverse benchmarks using four MLLMs demonstrate that our framework significantly outperforms various baselines in low-resource settings, achieving robust adaptation with minimal annotation costs.

Figures (4)

• Figure 1: Problem setup and performance comparison. (a) Unlike methods requiring large labeled datasets, our approach, VIOLA, strategically selects a minimal subset of informative samples for expert annotation while leveraging abundant unlabeled videos via pseudo-labeling. This constructs a hybrid pool from which the model retrieves relevant demonstrations for inference. (b) Performance comparison with Naive ICL (which randomly selects samples for annotation), averaged across seven classification datasets. VIOLA achieves robust adaptation with significantly reduced annotation costs.
• Figure 2: Overview of our proposed framework. The pipeline consists of three stages: 1. Selective Annotation: We acquire expert labels for a small, informative subset ($\mathcal{D}_L$) using density-uncertainty-weighted sampling. 2. Pseudo-Annotation: We generate high-confidence pseudo-labels via in-context pseudo-annotation to construct a hybrid pool $\mathcal{D}_H$. 3. Inference: We predict final answers via confidence-aware retrieval and prompting during inference.
• Figure 3: Performance trends under varying oracle annotation budgets. We compare our framework against baselines using Qwen2-VL-7B, varying the labeled pool size from 20 to 100 samples.
• Figure 4: Qualitative results on UCF-Crimes and EgoSurgery.