Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluation

TL;DR

<3-5 sentence high-level summary> The paper tackles amodal completion by reframing it as a planning-first problem, separating semantic reasoning from pixel synthesis through a Collaborative Multi-Agent Reasoning Framework. It introduces a closed-loop Chain-of-Thought verification mechanism and a Diverse Hypothesis Generator to address reasoning robustness and semantic ambiguity, respectively. To overcome evaluation misalignment, it presents the MAC-Score, with MAC-Completeness and MAC-Consistency, shown to align closely with human judgments. Extensive experiments on open-world and synthetic benchmarks demonstrate state-of-the-art performance and robust human-aligned evaluation, albeit with higher computational costs due to multi-agent reasoning.

Abstract

Amodal completion, the task of inferring invisible object parts, faces significant challenges in maintaining semantic consistency and structural integrity. Prior progressive approaches are inherently limited by inference instability and error accumulation. To tackle these limitations, we present a Collaborative Multi-Agent Reasoning Framework that explicitly decouples Semantic Planning from Visual Synthesis. By employing specialized agents for upfront reasoning, our method generates a structured, explicit plan before pixel generation, enabling visually and semantically coherent single-pass synthesis. We integrate this framework with two critical mechanisms: (1) a self-correcting Verification Agent that employs Chain-of-Thought reasoning to rectify visible region segmentation and identify residual occluders strictly within the Semantic Planning phase, and (2) a Diverse Hypothesis Generator that addresses the ambiguity of invisible regions by offering diverse, plausible semantic interpretations, surpassing the limited pixel-level variations of standard random seed sampling. Furthermore, addressing the limitations of traditional metrics in assessing inferred invisible content, we introduce the MAC-Score (MLLM Amodal Completion Score), a novel human-aligned evaluation metric. Validated against human judgment and ground truth, these metrics establish a robust standard for assessing structural completeness and semantic consistency with visible context. Extensive experiments demonstrate that our framework significantly outperforms state-of-the-art methods across multiple datasets. Our project is available at: https://fanhongxing.github.io/remac-page.

Figures (7)

• Figure 1: Our framework tackles complex occlusions through these key capabilities: (1) Structural & Semantic Reasoning, which recovers geometric continuity (e.g., hidden limbs) and contextual details (e.g., text) beyond pixel clues; and (2) Diverse Hypothesis Generation, which models the multimodal nature of invisible regions (e.g., diverse plushie states). Furthermore, we introduce (3) the MAC-Score, a human-aligned evaluation metric. As shown in the bottom-right, it resolves the mismatch where incomplete results are favored by traditional metrics, providing a robust standard for amodal completion.
• Figure 2: Common failure modes of progressive methods. (a) Inference Instability: The progressive process often terminates prematurely due to a lack of global planning, resulting in incomplete or truncated objects. (b) Error Accumulation: Early-stage errors propagate and amplify through iterative steps, causing structural inconsistencies and artifacts.
• Figure 3: Comparison of amodal completion paradigms. (a) Progressive Strategy: Iterative expansion ($\times N$) is vulnerable to error accumulation. (b) Our Framework: A holistic "reason-then-synthesize" approach. By determining the comprehensive plan upfront, we achieve robust single-pass synthesis, ensuring global consistency without iterative instability.
• Figure 4: Overview of the proposed Closed-Loop Collaborative Multi-Agent Reasoning Framework. The pipeline decouples semantic planning from visual synthesis through three stages: (1) Holistic Collaborative Reasoning: A coalition of agents synergizes to parse the scene's geometry, forming an initial spatial plan. (2) Closed-Loop Verification: A self-correcting mechanism where a Verification Agent scrutinizes the initial plan to correct segmentation errors and identify residual occluders. (3) Hypothesis Generation: The Hypothesis Agent generates multiple semantic descriptions for the invisible regions to capture diverse plausible interpretations. Finally, the Inpainting Agent executes the verified plan to synthesize the high-fidelity amodal result in a single pass.
• Figure 5: Comparison between traditional metrics and our MAC-Score. Traditional metrics fail to reflect human perception by assigning perfect scores to the incomplete Prediction A while penalizing the plausible Prediction C due to pixel deviations. In contrast, our MAC-Score correctly identifies Prediction C as the superior result with high consistency, aligning with human judgment.