Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MACD: Model-Aware Contrastive Decoding via Counterfactual Data

Qixin Xiao, Kun Zhou

TL;DR

MACD introduces a training-free, model-aware counterfactual decoding framework for Video-LLMs. By optimizing object- and frame-level masks via the model’s own loss, MACD creates targeted counterfactual inputs that reveal where hallucination stems, and integrates these into contrastive decoding to promote evidence-grounded tokens while suppressing unsupported ones. Across four benchmarks and multiple backbones, MACD consistently reduces object-level and global hallucinations and improves accuracy, with robust performance even under compute-constrained settings. The approach is lightweight, requiring only an additional forward pass for a counterfactual view and offering plug-and-play compatibility with other decoding strategies, making it practical for real-world deployment.

Abstract

Video language models (Video-LLMs) are prone to hallucinations, often generating plausible but ungrounded content when visual evidence is weak, ambiguous, or biased. Existing decoding methods, such as contrastive decoding (CD), rely on random perturbations to construct contrastive data for mitigating hallucination patterns. However, such a way is hard to control the visual cues that drive hallucination or well align with model weaknesses. We propose Model-aware Counterfactual Data based Contrastive Decoding (MACD), a new inference strategy that combines model-guided counterfactual construction with decoding. Our approach uses the Video-LLM's own feedback to identify object regions most responsible for hallucination, generating targeted counterfactual inputs at the object level rather than arbitrary frame or temporal modifications. These model-aware counterfactual data is then integrated into CD to enforce evidence-grounded token selection during decoding. Experiments on EventHallusion, MVBench, Perception-test and Video-MME show that MACD consistently reduces hallucination while maintaining or improving task accuracy across diverse Video-LLMs, including Qwen and InternVL families. The method is especially effective in challenging scenarios involving small, occluded, or co-occurring objects. Our code and data will be publicly released.

MACD: Model-Aware Contrastive Decoding via Counterfactual Data

TL;DR

MACD introduces a training-free, model-aware counterfactual decoding framework for Video-LLMs. By optimizing object- and frame-level masks via the model’s own loss, MACD creates targeted counterfactual inputs that reveal where hallucination stems, and integrates these into contrastive decoding to promote evidence-grounded tokens while suppressing unsupported ones. Across four benchmarks and multiple backbones, MACD consistently reduces object-level and global hallucinations and improves accuracy, with robust performance even under compute-constrained settings. The approach is lightweight, requiring only an additional forward pass for a counterfactual view and offering plug-and-play compatibility with other decoding strategies, making it practical for real-world deployment.

Abstract

Video language models (Video-LLMs) are prone to hallucinations, often generating plausible but ungrounded content when visual evidence is weak, ambiguous, or biased. Existing decoding methods, such as contrastive decoding (CD), rely on random perturbations to construct contrastive data for mitigating hallucination patterns. However, such a way is hard to control the visual cues that drive hallucination or well align with model weaknesses. We propose Model-aware Counterfactual Data based Contrastive Decoding (MACD), a new inference strategy that combines model-guided counterfactual construction with decoding. Our approach uses the Video-LLM's own feedback to identify object regions most responsible for hallucination, generating targeted counterfactual inputs at the object level rather than arbitrary frame or temporal modifications. These model-aware counterfactual data is then integrated into CD to enforce evidence-grounded token selection during decoding. Experiments on EventHallusion, MVBench, Perception-test and Video-MME show that MACD consistently reduces hallucination while maintaining or improving task accuracy across diverse Video-LLMs, including Qwen and InternVL families. The method is especially effective in challenging scenarios involving small, occluded, or co-occurring objects. Our code and data will be publicly released.
Paper Structure (68 sections, 9 equations, 1 figure, 14 tables)

This paper contains 68 sections, 9 equations, 1 figure, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models.
  4. Hallucination in LLMs.
  5. Contrastive Decoding.
  6. Preliminary
  7. Video Large Language Models.
  8. Contrastive Decoding.
  9. Our Target.
  10. Method
  11. Counterfactual data augmentation
  12. Object Detection
  13. Masked Data Collection
  14. Model-aware contrastive data optimization
  15. Contrastive decoding
  16. ...and 53 more sections

Figures (1)

  • Figure 1: The overview of our proposed method. It consists of the model-aware counterfactual data augmentation using object-level and frame-level masks that can be optimized via the gradients from the VLM, and the contrastive decoding strategy to boost the probability of important tokens relevant to the masked part in the video.