Multi-agent Undercover Gaming: Hallucination Removal via Counterfactual Test for Multimodal Reasoning

TL;DR

The paper introduces Multi-agent Undercover Gaming (MUG), a protocol that detects hallucinating agents in multimodal reasoning by using counterfactual test scenarios where an edited image $I^-$ challenges agents relying on $I^+$. By reframing MAD as an undercover detection game with active reasoning and cross-evidence through dynamic counterfactuals, MUG provides ground-truth signals for agent reliability and robust, crowd-powered reasoning. Empirical results across multiple multimodal benchmarks show that MUG improves reasoning accuracy and hallucination detection, narrows performance gaps between model scales, and offers insights into the dynamics of game rounds and termination. The approach advances reliable multimodal reasoning in LLMs with practical implications for safety-critical AI systems and collaborative intelligence.

Abstract

Hallucination continues to pose a major obstacle in the reasoning capabilities of large language models (LLMs). Although the Multi-Agent Debate (MAD) paradigm offers a promising solution by promoting consensus among multiple agents to enhance reliability, it relies on the unrealistic assumption that all debaters are rational and reflective, which is a condition that may not hold when agents themselves are prone to hallucinations. To address this gap, we introduce the Multi-agent Undercover Gaming (MUG) protocol, inspired by social deduction games like "Who is Undercover?". MUG reframes MAD as a process of detecting "undercover" agents (those suffering from hallucinations) by employing multimodal counterfactual tests. Specifically, we modify reference images to introduce counterfactual evidence and observe whether agents can accurately identify these changes, providing ground-truth for identifying hallucinating agents and enabling robust, crowd-powered multimodal reasoning. MUG advances MAD protocols along three key dimensions: (1) enabling factual verification beyond statistical consensus through counterfactual testing; (2) introducing cross-evidence reasoning via dynamically modified evidence sources instead of relying on static inputs; and (3) fostering active reasoning, where agents engage in probing discussions rather than passively answering questions. Collectively, these innovations offer a more reliable and effective framework for multimodal reasoning in LLMs. The source code can be accessed at https://github.com/YongLD/MUG.git.

Figures (10)

• Figure 1: Overview of the Counterfactual Undercover Debate Framework, illustrating the dynamic interactions between agents, the current game state, and the decision-making process influenced by counterfactual information. The diagram highlights the roles of normal and undercover agents, as well as the flow of information and reasoning throughout the game.
• Figure 2: Comparison of Chain-of-thought (CoT), Multi-agent Debate (MAD), and our proposed Multi-agent Undercover Gaming (MUG). Q: input question, $I^+$: input image, $I^-$: edited counterfactual image, A: MLLM agent, R: output response.
• Figure 3: Illustration of counterfactual editing constraints, including visual similarity, semantic change, and naturalness.
• Figure 4: Performance comparison for different models across hallucination categories in the POPE and HallusionBench datasets.
• Figure 5: Ablation results demonstrating the impact of counterfactual editing and undercover mechanisms on performance across three benchmarks.