LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotations

TL;DR

This work introduces LAION-SG, a large-scale scene-graph annotated dataset built atop LAION-Aesthetic V2 to enable robust compositional image generation. It pairs SGs with images and trains a SG-aware foundation model, SDXL-SG, by integrating a graph-neural-network-based SG encoder into Stable Diffusion XL, yielding superior performance on complex scenes. A new CompSGen Bench provides standardized evaluation using SG-IoU, Entity-IoU, and Relation-IoU alongside Fréchet Inception Distance and CLIP scores, with experiments showing substantial gains over text-only prompts and prior SG2IM methods. The dataset, model, and benchmark collectively advance the data- and model-level capabilities for SG-guided image synthesis, with potential impacts across content creation and AI-assisted perception, while acknowledging automation and bias considerations.

Abstract

Recent advances in text-to-image (T2I) generation have shown remarkable success in producing high-quality images from text. However, existing T2I models show decayed performance in compositional image generation involving multiple objects and intricate relationships. We attribute this problem to limitations in existing datasets of image-text pairs, which lack precise inter-object relationship annotations with prompts only. To address this problem, we construct LAION-SG, a large-scale dataset with high-quality structural annotations of scene graphs (SG), which precisely describe attributes and relationships of multiple objects, effectively representing the semantic structure in complex scenes. Based on LAION-SG, we train a new foundation model SDXL-SG to incorporate structural annotation information into the generation process. Extensive experiments show advanced models trained on our LAION-SG boast significant performance improvements in complex scene generation over models on existing datasets. We also introduce CompSG-Bench, a benchmark that evaluates models on compositional image generation, establishing a new standard for this domain. Our annotations with the associated processing code, the foundation model and the benchmark protocol are publicly available at https://github.com/mengcye/LAION-SG.

Figures (13)

• Figure 1: The generated images given by the text-to-image (T2I) model SDXL podell2023sdxlimprovinglatentdiffusion and SDXL-SG, a T2I model with structural annotation guidance given different numbers of relations. For one or two relations, both models can generate accurately. When dealing with three or four relations, the T2I model fails to generate the relations "hold" and "facing". For cases with more than four relations, the limitations of the T2I model become more pronounced. In (e) and (f), three and two relations are incorrectly generated, respectively. In comparison, SDXL-SG accurately captures the relations as shown in the generated images.
• Figure 2: The construction pipeline of LAION-SG dataset. 1) Identify the objects in the image and assign a unique ID to each. 2) The attributes must be abstract adjectives and should not include specific objects. Each object may have one or more attributes. 3) The relations between objects should be as specific as possible, avoiding simple relations. Use more precise verbs, minimizing repetition. 4) For people, label the object as "person" and include attributes such as gender and age. Avoid anthropomorphism or associations, and provide an objective description of what is observed in the image.
• Figure 3: The annotation length and accuracy characteristics of LAION-SG compared to the LAION-Aesthetics. Compared to text, the scene graph, as a more compact form, has a longer length and its accuracy is more concentrated in high-scoring areas. This suggests that our LAION-SG annotation more accurately reflects the image information and contains richer semantics.
• Figure 4: The annotation distribution of LAION-SG. (a) The length of the scene graph lies in a wide range. Our annotation provides more specific information compared to single-word descriptions, while also avoiding the inefficiency in model learning caused by excessively lengthy annotations. (b) The top 10 relations and attributes represent only a small percentage of the total distribution, indicating that LAION-SG covers a highly diverse range of annotations, showcasing its large scale and open vocabulary.
• Figure 5: Visual comparison on LAION-SG. The compared methods include T2I model (SDXL podell2023sdxlimprovinglatentdiffusion) and SG2IM models (SGDiff Yang2022DiffusionBasedSG and SG-Adapter Shen2024SGAdapterET). The first column shows the original caption from LAION-Aesthetics. The second column displays the scene graph from our LAION-SG. The last five columns show ground truth images and images generated by different models. Objects or relations are highlighted with the same color in scene graphs and generated images to show SDXL-SG successfully captures the complex scenes.