SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editing

TL;DR

SimGraph addresses the fragmentation between image generation and editing by unifying them under a single scene graph–driven model. It combines token-based generation conditioned on scene-graph captions with diffusion-based editing guided by joint source/target prompts, ensuring spatial coherence and semantic alignment. The framework leverages scene-graph extraction, a VAR–CLIP generation path, and LEDIT++ editing, delivering higher fidelity and efficiency than prior scene-graph methods. This unified approach enables precise control over object interactions and layouts, with practical impact for flexible GenAI workflows and real-time editing scenarios.

Abstract

Recent advancements in Generative Artificial Intelligence (GenAI) have significantly enhanced the capabilities of both image generation and editing. However, current approaches often treat these tasks separately, leading to inefficiencies and challenges in maintaining spatial consistency and semantic coherence between generated content and edits. Moreover, a major obstacle is the lack of structured control over object relationships and spatial arrangements. Scene graph-based methods, which represent objects and their interrelationships in a structured format, offer a solution by providing greater control over composition and interactions in both image generation and editing. To address this, we introduce SimGraph, a unified framework that integrates scene graph-based image generation and editing, enabling precise control over object interactions, layouts, and spatial coherence. In particular, our framework integrates token-based generation and diffusion-based editing within a single scene graph-driven model, ensuring high-quality and consistent results. Through extensive experiments, we empirically demonstrate that our approach outperforms existing state-of-the-art methods.

Figures (3)

• Figure 1: Illustration of SimGraph, which shares the same strategy for scene graph extraction using MLLM (e.g., Qwen-VL bai2023qwen) (introduced in Sec. \ref{['subsec:sg_extract']}). In addition, our framework simultaneously integrates token-based image generation (introduced in Sec. \ref{['subsec:img_gen']}) and diffusion model for image editing (introduced in Sec. \ref{['subsec:img_edit']}).
• Figure 2: Illustration of image generation and editing from scene graphs using our framework. The left side shows the input image with its corresponding scene graph. The generated image on the right demonstrates the model's ability to faithfully recreate the scene from the extracted scene graph. The edited scene graph highlights the modifications made, such as replacing the "bear" with a "wolf" in the forest and adding a "tiger". Best viewed in color and zoomed in.
• Figure 3: Illustration of failure cases. Best viewed in color and zoomed in.