Seeing Beyond the Scene: Enhancing Vision-Language Models with Interactional Reasoning

TL;DR

This work tackles the limited interaction reasoning of vision-language models caused by spatially focused scene graphs. It introduces ISGR, a framework comprising a summarize-and-align graph construction, an interactional chain-of-thought reasoning process (ICoT), and a long-term memory reinforcement (LTMR) mechanism to generalize interaction understanding. By building spatial and abstract graphs, extracting and refining interactions, and reinforcing memory through reward-guided training, ISGR achieves strong gains on interaction-heavy benchmarks while reducing data requirements. The approach demonstrates robust generalization to unseen scenes and interactions, suggesting that structured relational modeling with memory can substantially enhance multimodal reasoning in VLMs.

Abstract

Traditional scene graphs primarily focus on spatial relationships, limiting vision-language models' (VLMs) ability to reason about complex interactions in visual scenes. This paper addresses two key challenges: (1) conventional detection-to-construction methods produce unfocused, contextually irrelevant relationship sets, and (2) existing approaches fail to form persistent memories for generalizing interaction reasoning to new scenes. We propose Interaction-augmented Scene Graph Reasoning (ISGR), a framework that enhances VLMs' interactional reasoning through three complementary components. First, our dual-stream graph constructor combines SAM-powered spatial relation extraction with interaction-aware captioning to generate functionally salient scene graphs with spatial grounding. Second, we employ targeted interaction queries to activate VLMs' latent knowledge of object functionalities, converting passive recognition into active reasoning about how objects work together. Finally, we introduce a lone-term memory reinforcement learning strategy with a specialized interaction-focused reward function that transforms transient patterns into long-term reasoning heuristics. Extensive experiments demonstrate that our approach significantly outperforms baseline methods on interaction-heavy reasoning benchmarks, with particularly strong improvements on complex scene understanding tasks. The source code can be accessed at https://github.com/open_upon_acceptance.

Figures (5)

• Figure 1: Examples showing how our interaction-augmented scene graphs enhance reasoning on dynamic interactions. Spatial: a conventional spatial-only scene graph misinterprets the situation as merely "baseball in front of player". Interaction: our approach correctly identifies the functional relationship "player catch baseball", enabling more accurate answer to the query "Who is trying to catch the baseball?".
• Figure 2: Overview of our Interaction-augmented Scene Graph Reasoning (ISGR) framework: (a) Summarize-and-Align Graph Construction transforms input images through Scene Graph Initialization and ICOT, progressively creating Spatial, Abstract, and Interaction Graphs with relevance, focus, and disambiguation constraints; (b) Long-Term Memory Reinforcement combines ISGR(SFT) and ISGR(SFT+IRR) models with Interaction Reasoning Reinforcement to enhance interaction reasoning capabilities on complex visual questions.
• Figure 3: Performance comparison on scene reasoning benchmarks. Our proposed models (ISGR(SFT) and ISGR(SFT+IRR)) consistently outperform baseline models (LLaVA and LLaVA-IRR) across diverse benchmarks measuring different aspects of scene understanding.
• Figure 4: Category Performance Comparison on SEEDBench. ISGR(SFT+IRR) shows significant improvements over the LLaVA-v1.5 baseline across most categories.
• Figure 5: A case study for our proposed ISGR framework: (Left) Limitations of spatial reasoning where models provide contradictory answers based solely on proximity; (Medium) Scene graph focusing enables accurate identification of functional interactions; (Right) Long-term memory reinforcement enhances subtle relationship identification and generalization to novel scenes, including unseen relationship types in generated images.