Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models

Wenbin Xing, Quanxing Zha, Lizheng Zu, Mengran Li, Ming Li, Junchi Yan

TL;DR

We tackle the challenge of compositional hallucinations in video multimodal LLMs by introducing OmniVCHall, a benchmark that jointly evaluates isolated and compositional failures across eight fine-grained types (including a novel camera-type) with adversarial answer options, real- and AI-generated videos, and YNQA/MCQA tasks. To mitigate these multi-factor errors, we propose TriCD, a triple-pathway decoding framework that combines an adaptive perturbation controller for dynamic negative samples with a saliency-guided enhancement module for stronger grounded evidence; both components are trained via policy gradient reinforcement learning. Empirical results across 39 VLLMs reveal substantial performance gaps relative to human accuracy, especially under compositional queries and adversarial options. TriCD yields consistent, scalable improvements (e.g., Avg gains of $+9.61\%$ on Qwen3-VL-Instruct-8B and $+12.03\%$ on VideoLLaMA3-7B) and outperforms existing CD baselines, underscoring the value of context-aware negative suppression and spatiotemporal grounding in mitigating compositional hallucinations. The authors provide data and code at https://github.com/BMRETURN/OmniVCHall to facilitate further research and benchmarking.

Abstract

Current research on video hallucination mitigation primarily focuses on isolated error types, leaving compositional hallucinations, arising from incorrect reasoning over multiple interacting spatial and temporal factors largely underexplored. We introduce OmniVCHall, a benchmark designed to systematically evaluate both isolated and compositional hallucinations in video multimodal large language models (VLLMs). OmniVCHall spans diverse video domains, introduces a novel camera-based hallucination type, and defines a fine-grained taxonomy, together with adversarial answer options (e.g., "All are correct" and "None of the above") to prevent shortcut reasoning. The evaluations of 39 representative VLLMs reveal that even advanced models (e.g., Qwen3-VL and GPT-5) exhibit substantial performance degradation. We propose TriCD, a contrastive decoding framework with a triple-pathway calibration mechanism. An adaptive perturbation controller dynamically selects distracting operations to construct negative video variants, while a saliency-guided enhancement module adaptively reinforces grounded token-wise visual evidences. These components are optimized via reinforcement learning to encourage precise decision-making under compositional hallucination settings. Experimental results show that TriCD consistently improves performance across two representative backbones, achieving an average accuracy improvement of over 10%. The data and code can be find at https://github.com/BMRETURN/OmniVCHall.

Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models

TL;DR

We tackle the challenge of compositional hallucinations in video multimodal LLMs by introducing OmniVCHall, a benchmark that jointly evaluates isolated and compositional failures across eight fine-grained types (including a novel camera-type) with adversarial answer options, real- and AI-generated videos, and YNQA/MCQA tasks. To mitigate these multi-factor errors, we propose TriCD, a triple-pathway decoding framework that combines an adaptive perturbation controller for dynamic negative samples with a saliency-guided enhancement module for stronger grounded evidence; both components are trained via policy gradient reinforcement learning. Empirical results across 39 VLLMs reveal substantial performance gaps relative to human accuracy, especially under compositional queries and adversarial options. TriCD yields consistent, scalable improvements (e.g., Avg gains of on Qwen3-VL-Instruct-8B and on VideoLLaMA3-7B) and outperforms existing CD baselines, underscoring the value of context-aware negative suppression and spatiotemporal grounding in mitigating compositional hallucinations. The authors provide data and code at https://github.com/BMRETURN/OmniVCHall to facilitate further research and benchmarking.

Abstract

Current research on video hallucination mitigation primarily focuses on isolated error types, leaving compositional hallucinations, arising from incorrect reasoning over multiple interacting spatial and temporal factors largely underexplored. We introduce OmniVCHall, a benchmark designed to systematically evaluate both isolated and compositional hallucinations in video multimodal large language models (VLLMs). OmniVCHall spans diverse video domains, introduces a novel camera-based hallucination type, and defines a fine-grained taxonomy, together with adversarial answer options (e.g., "All are correct" and "None of the above") to prevent shortcut reasoning. The evaluations of 39 representative VLLMs reveal that even advanced models (e.g., Qwen3-VL and GPT-5) exhibit substantial performance degradation. We propose TriCD, a contrastive decoding framework with a triple-pathway calibration mechanism. An adaptive perturbation controller dynamically selects distracting operations to construct negative video variants, while a saliency-guided enhancement module adaptively reinforces grounded token-wise visual evidences. These components are optimized via reinforcement learning to encourage precise decision-making under compositional hallucination settings. Experimental results show that TriCD consistently improves performance across two representative backbones, achieving an average accuracy improvement of over 10%. The data and code can be find at https://github.com/BMRETURN/OmniVCHall.
Paper Structure (66 sections, 10 equations, 18 figures, 18 tables, 3 algorithms)

This paper contains 66 sections, 10 equations, 18 figures, 18 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. OmniVCHall Benchmark
  3. Taxonomy Design
  4. Fine-grained Hallucination Type.
  5. Hierarchical Question Complexity.
  6. Diverse Video Source.
  7. Multi-faceted QA Task.
  8. Benchmark Construction
  9. Video Collection and Curation.
  10. QA Generation with Adversarial Injection.
  11. Data Statistics
  12. TriCD Framework
  13. Original Pass & State Extraction
  14. Negative Pass & Positive Pass
  15. Adaptive Perturbation Controller
  16. ...and 51 more sections

Figures (18)

  • Figure 1: (a) Existing VLLMs frequently struggle with cross-modal video understanding tasks involving compositional hallucinations. (b) Extensive evaluation across 39 models reveals a pronounced accuracy drop ($\downarrow$5.71% and $\downarrow$9.32%) when transitioning from single-factor hallucination queries to compositional ones.
  • Figure 2: Overview of the OmniVCHall benchmark. (a) shows a hierarchical structure. (b) shows a three-step pipeline. (c) utilizes adversarial answer options (e.g., "All are correct” and "None of the above”) to discourage shortcut reasoning.
  • Figure 3: Statistical analysis of OmniVCHall. From left to right: distribution of the four sub-tasks across domains, histograms of video durations, and word count distributions highlighting the complexity of compositional tasks.
  • Figure 4: TriCD framework. (a) employs a three-step process that uses contrastive decoding to refine final predictions. (b) dynamically selects the most contextually relevant tools from a bank of eight video perturbation tools via cross-attention to construct a negative sample. (c) fuses spatial (DINOv3) and temporal (motion) saliency maps to reweight vision tokens, anchoring the positive pass to critical evidence.
  • Figure 5: Comparative performance of the leading representatives from each model family. Human performance is highlighted at the top to establish a ceiling for evaluating current VLLM capabilities.
  • ...and 13 more figures