Spatial Knowledge Graph-Guided Multimodal Synthesis

TL;DR

This paper addresses the limited spatial understanding of Multimodal Large Language Models by introducing SKG2Data, a framework that uses Spatial Knowledge Graphs to guide the targeted synthesis of spatially coherent images and textual data. The approach combines SKG generation with diffusion-based image synthesis and LLM-guided QA data, facilitating automated, scalable creation of a spatially aware multimodal dataset (SKG2Data) and a holdout benchmark (SKG2Data-Holdout). Experiments show that fine-tuning LLaVA-1.6 and Llama-3.2-Vision with SKG2Data yields notable improvements on spatial benchmarks (KG2Data-Holdout, COCO-Spatial, MMVP) while maintaining general visual capabilities. Key findings reveal the critical role of directional knowledge, the generalization benefits of combining relation types, and the positive impact of higher object density on fine-grained spatial tasks, highlighting the practical potential of knowledge-guided data synthesis for spatial intelligence in AI systems.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities; however, their spatial perception abilities remain a notable limitation. To address this challenge, multimodal data synthesis offers a promising solution. Yet, ensuring that synthesized data adhere to spatial common sense is a non-trivial task. Our approach addresses this critical gap by providing a systematic framework for generating spatially coherent data. In this work, we introduce SKG2DATA, a novel multimodal synthesis approach guided by spatial knowledge graphs, grounded in the concept of knowledge-to-data generation. SKG2DATA employs an automated pipeline for constructing Spatial Knowledge Graph (SKG) that effectively captures human-like spatial cognition, including directional and distance relationships. These structured representations then serve as precise guidance for our integrated synthesis pipeline, where a diffusion model generates spatially-consistent images while a MLLM produces corresponding textual descriptions. The automated construction of SKG enables scalable generation of diverse yet realistic spatial configurations, overcoming the limitations of manual data collection and annotation. Extensive experiments demonstrate that data synthesized from diverse types of spatial knowledge, including direction and distance, enhance the spatial perception and reasoning abilities of MLLMs markedly, albeit with a slight cost to their general capabilities. We hope that the idea of knowledge-based data synthesis can advance the development of spatial intelligence. Code is available at https://github.com/zjunlp/Knowledge2Data.

Figures (8)

• Figure 1: The Spatial Knowledge Graph (SKG) serves as a structured representation of object attributes and spatial relationships. Guided by the SKG, SKG2Data generates images and multimodal data, grounded in the concept of knowledge-to-data generation.
• Figure 2: A comprehensive overview of the our framework. Our framework consists of two core modules: Spatial KG Generation and Multimodal Data Synthesis. The Spatial KG Generation module generates an intermediate representation, the Spatial KG, which guides the synthesis of multimodal data. The Multimodal Data Synthesis module is tasked with generating image data and their corresponding textual data.
• Figure 3: Distribution of top 15 objects and spatial relationships. There are a total of 974 objects and 95 relationships.
• Figure 4: Training data cases featuring diverse spatial relationships, synthesized through our proposed method.
• Figure 5: The result of removing specific positional relationship data with the same amount of data.