Heterogeneous Image GNN: Graph-Conditioned Diffusion for Image Synthesis

TL;DR

This work tackles conditioning diffusion-based image synthesis on heterogeneous, graph-structured data by introducing Heterogeneous Image Graphs (HIG), which jointly model images and conditioning signals as interconnected graphs. A magnitude-preserving MP-GNN (HIGnn) processes the HIG, and its latent conditioning is incorporated into a frozen EDM2 model via a ControlNet-style encoder, enabling end-to-end diffusion with graph-conditioned signals. The approach achieves state-of-the-art results on layout-to-image and mask-to-image tasks (e.g., COCO-Stuff and Visual Genome) and demonstrates precise local control over attributes, objects, and relationships, while remaining computationally efficient relative to quadratic-attention baselines. This framework broadens the scope of conditioning signals for diffusion models, enabling flexible, multi-modal, and fine-grained generation across diverse datasets and conditioning modalities.

Abstract

We introduce a novel method for conditioning diffusion-based image synthesis models with heterogeneous graph data. Existing approaches typically incorporate conditioning variables directly into model architectures, either through cross-attention layers that attend to text latents or image concatenation that spatially restrict generation. However, these methods struggle to handle complex scenarios involving diverse, relational conditioning variables, which are more naturally represented as unstructured graphs. This paper presents Heterogeneous Image Graphs (HIG), a novel representation that models conditioning variables and target images as two interconnected graphs, enabling efficient handling of variable-length conditioning inputs and their relationships. We also propose a magnitude-preserving GNN that integrates the HIG into the existing EDM2 diffusion model using a ControlNet approach. Our approach improves upon the SOTA on a variety of conditioning inputs for the COCO-stuff and Visual Genome datasets, and showcases the ability to condition on graph attributes and relationships represented by edges in the HIG.

Figures (18)

• Figure 1: (a) Our representation enables flexible conditioning for graph-to-image generation by modeling objects, attributes, and relationships as a graph. This integrates with a pixel grid, where pixels act as nodes connected to objects, defining spatial relationships. (b) Modifying conditions at the node and or edge level enables precise semantic control over the generation process.
• Figure 2: (a) Overview of the proposed architecture. The HIG is encoded into a latent representation through a MP-GNN which is then used as a condition $c_f$ in a ControlNet. (b) Details of the MP-GNN module. Note: HMP is shorthand for heterogenous magnitude preserving operations applied across all nodes.
• Figure 3: Comparison with SOTA and reference on COCO-stuff.
• Figure 4: Generations from the same seed demonstrate HIG's ability to control size, quantity and position of objects.
• Figure 5: HIG enables precise, localized control over semantic conditions, including attributes, objects, and their relationships.