SceneProp: Combining Neural Network and Markov Random Field for Scene-Graph Grounding

TL;DR

SceneProp reframes scene-graph grounding as a MAP inference problem on a Markov Random Field, enabling global, relational reasoning across all object candidates. By integrating neural unary and pairwise potentials with a differentiable Belief Propagation framework, it achieves end-to-end training and principled inference, overcoming prior methods' degradation on larger, more descriptive graphs. Across VG-FO, VG150, COCO-Stuff, and GQA, SceneProp not only outperforms baselines but also improves as the query graph grows in complexity, illustrating robust relational reasoning in long-tail, multi-object scenes. The work highlights a practical path for scalable, accurate scene-graph grounding with strong relational context, and it outlines future directions toward open-vocabulary grounding and graph parsing from natural language.

Abstract

Grounding complex, compositional visual queries with multiple objects and relationships is a fundamental challenge for vision-language models. While standard phrase grounding methods excel at localizing single objects, they lack the structural inductive bias to parse intricate relational descriptions, often failing as queries become more descriptive. To address this structural deficit, we focus on scene-graph grounding, a powerful but less-explored formulation where the query is an explicit graph of objects and their relationships. However, existing methods for this task also struggle, paradoxically showing decreased performance as the query graph grows -- failing to leverage the very information that should make grounding easier. We introduce SceneProp, a novel method that resolves this issue by reformulating scene-graph grounding as a Maximum a Posteriori (MAP) inference problem in a Markov Random Field (MRF). By performing global inference over the entire query graph, SceneProp finds the optimal assignment of image regions to nodes that jointly satisfies all constraints. This is achieved within an end-to-end framework via a differentiable implementation of the Belief Propagation algorithm. Experiments on four benchmarks show that our dedicated focus on the scene-graph grounding formulation allows SceneProp to significantly outperform prior work. Critically, its accuracy consistently improves with the size and complexity of the query graph, demonstrating for the first time that more relational context can, and should, lead to better grounding. Codes are available at https://github.com/keitaotani/SceneProp.

Figures (17)

• Figure 1: Compositional grounding with SceneProp. Existing methods fail due to many partial matches: (a) GPT-4o, using description as a proxy task for its poor coordinate output, and (b) Grounding DINO. (c) In contrast, our SceneProp succeeds by finding an object combination that satisfies the query graph.
• Figure 2: A simple example of an MRF.
• Figure 3: Overview of the SceneProp's pipeline. First, image features and bounding boxes are extracted (\ref{['sec:region_proposal']}). The bounding boxes are encoded using our proposed positional encoding (\ref{['sec:positional_encoding']}). Using these encoded positions and image features, the relationship features are extracted (\ref{['sec:relationship_feature']}). Then, the energy of the object label and relationship are calculated using the relationship feature. Using this energy and the given query graph, we construct and infer the Scene-MRF (\ref{['sec:inference_with_scene_mrf']}).
• Figure 4: Impact of query graph size (number of relationships) on grounding performance on the VG-FO dataset. SceneProp's performance improves with more descriptive queries, while both VL-MPAG and Grounding DINO (DINO) show declining recall.
• Figure 5: Mean recall over top-k frequent relationships on GQA. SceneProp's performance remains stable even when including rare relationship categories.