Graph Recognition via Subgraph Prediction

TL;DR

This work addresses the challenge of extracting graphs from images by proposing GraSP, a unified, transfer-friendly framework that treats graph generation as sequential subgraph prediction conditioned on the image. It reframes learning as a Markov decision process and replaces the value function with a binary subgraph classifier to enable efficient, termination-aware decoding that is independent of the graph encoding. The architecture combines a Graph Neural Network with a FiLM-conditioned CNN, trained via streaming data generation to handle variable-sized graphs and to scale. Across synthetic and real-world tasks, GraSP demonstrates transferable performance and generalization capabilities, highlighting its potential to unify visual graph recognition while revealing practical considerations for scalable training and future enhancements.

Abstract

Despite tremendous improvements in tasks such as image classification, object detection, and segmentation, the recognition of visual relationships, commonly modeled as the extraction of a graph from an image, remains a challenging task. We believe that this mainly stems from the fact that there is no canonical way to approach the visual graph recognition task. Most existing solutions are specific to a problem and cannot be transferred between different contexts out-of-the box, even though the conceptual problem remains the same. With broad applicability and simplicity in mind, in this paper we develop a method, \textbf{Gra}ph Recognition via \textbf{S}ubgraph \textbf{P}rediction (\textbf{GraSP}), for recognizing graphs in images. We show across several synthetic benchmarks and one real-world application that our method works with a set of diverse types of graphs and their drawings, and can be transferred between tasks without task-specific modifications, paving the way to a more unified framework for visual graph recognition.

Figures (6)

• Figure 1: Feed-forward architecture of our approach. The model first receives a graph as input and produces a graph embedding. This embedding is used to condition the image embedding on a particular graph. The prediction head uses this embedding, with an additional terminal flag, to predict whether a particular graph is a subgraph of the graph shown in the image.
• Figure 2: Training dynamics of several synthetic tasks of varying complexity on trees with six to nine nodes. Shaded areas correspond to the standard deviation of a window of 10 iterations, i.e., 10M samples for training and 1000 trajectories for evaluation. Experiments X NC and Y EC have been conducted on graphs with X node and Y edge colors. $k$ varies depending on the number of valid successor states.
• Figure 3: Training dynamics of several synthetic tasks of varying complexity on trees with $10$ to $15$ nodes. Shaded areas correspond to the standard deviation of a window of 10 iterations, i.e., 10M samples for training and 1000 trajectories for evaluation. Experiments X NC and Y EC have been conducted on graphs with X node and Y edge colors. $k$ varies depending on the number of valid successor states.
• Figure 4: Accuracy on the QM9 dataset.
• Figure 5: Training dynamics for graphs of size 6 to 9.