SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes

TL;DR

SceneAlign tackles faithful multimodal reasoning in complex visual scenes by grounding reasoning in scene graphs and training with contrastive, structure-aware negatives. It constructs positives anchored to a scene graph and generates hard negatives by four graph perturbations (swap, replace, shorten, overthink), selecting diverse, moderately hard negatives and optimizing with Direct Preference Optimization (DPO) to align reasoning with visual structure. Across seven benchmarks, SceneAlign improves reasoning faithfulness and answer accuracy, especially on multi-step and relation-sensitive tasks, demonstrating grounding-aware alignment as a practical way to reduce hallucinations and mis-groundings. The approach is model-agnostic and scalable, improving performance even for smaller backbones.

Abstract

Multimodal large language models often struggle with faithful reasoning in complex visual scenes, where intricate entities and relations require precise visual grounding at each step. This reasoning unfaithfulness frequently manifests as hallucinated entities, mis-grounded relations, skipped steps, and over-specified reasoning. Existing preference-based approaches, typically relying on textual perturbations or answer-conditioned rationales, fail to address this challenge as they allow models to exploit language priors to bypass visual grounding. To address this, we propose SceneAlign, a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. By identifying reasoning-critical nodes and perturbing them through four targeted strategies that mimic typical grounding failures, SceneAlign constructs hard negative rationales that remain linguistically plausible but are grounded in inaccurate visual facts. These contrastive pairs are used in Direct Preference Optimization to steer models toward fine-grained, structure-faithful reasoning. Across seven visual reasoning benchmarks, SceneAlign consistently improves answer accuracy and reasoning faithfulness, highlighting the effectiveness of grounding-aware alignment for multimodal reasoning.

Figures (8)

• Figure 1: Motivation for structure-aware supervision. Under structure-agnostic supervision, reasoning errors that ignore, hallucinate, or loosely associate visual structure are indistinguishable at the answer level. Structure-aware supervision (SceneAlign) perturbs scene-graph elements to generate interpretable and controllable negative examples, making grounding failures observable and localizable.
• Figure 2: Framework overview. The SceneAlign pipeline first generates scene-graph–grounded positives (Sec. \ref{['sec:positive']}), applies four graph perturbations (swap, replace, shorten, overthink) to create negatives (Sec. \ref{['sec:negatives']}), filters for diversity (Sec. \ref{['sec:select']}), and finally performs DPO alignment (Sec. \ref{['sec:dpo']}).
• Figure 3: Sensitivity to Jaccard bounds $\gamma_\ell$ and $\gamma_u$, Performance shows an inverted U-shape with optimal performance at $\gamma_\ell = 0.3$ and $\gamma_u = 0.7$.
• Figure 4: Example Image from Case Study.
• Figure 5: Prompt for structured scene graph generation.