Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Grounding

TL;DR

SOC tackles data scarcity in open-world vision tasks by generating a vast, highly annotated pool of object segments and composing them into diverse scenes with 3D-layout and camera augmentations plus photorealistic harmonization. Its object-centric approach yields pixel-perfect masks, boxes, and dense referring expressions, enabling competitive open-vocabulary detection, visual grounding, and instance segmentation at scale. Across tasks and data regimes, SOC outperforms existing synthetic pipelines and provides additive gains when combined with real datasets, while also enabling targeted diagnostics like intra-class referring expressions. The work demonstrates practical impact by improving performance in low-data settings and offering a versatile, controllable data synthesis framework for scalable, accurate scene understanding.

Abstract

Visual grouping -- operationalized through tasks such as instance segmentation, visual grounding, and object detection -- enables applications ranging from robotic perception to photo editing. These fundamental problems in computer vision are powered by large-scale, painstakingly annotated datasets. Despite their impact, these datasets are costly to build, biased in coverage, and difficult to scale. Synthetic datasets offer a promising alternative but struggle with flexibility, accuracy, and compositional diversity. We introduce Synthetic Object Compositions (SOC), an accurate and scalable data synthesis pipeline via a novel object-centric composition strategy. It composes high-quality synthetic object segments into new images using 3D geometric layout augmentation and camera configuration augmentation with generative harmonization and mask-area-weighted blending, yielding accurate and diverse masks, boxes, and referring expressions. Models trained on just 100K of our synthetic images outperform those trained on larger real datasets (GRIT 20M, V3Det 200K) and synthetic pipelines (Copy-Paste, X-Paste, SynGround, SegGen) by +24-36% -- achieving +10.9 AP on LVIS and +8.4 NAcc on gRefCOCO. Beyond the general open-vocabulary setup, SOC also enables controllable dataset construction for different use cases and boosts performance in both low-data and closed-vocabulary scenarios. Augmenting LVIS and COCO with synthetic object segments delivers strong performance across different real-data scales and yields even greater improvements under extremely limited real-data conditions, including +6.59 AP on a 1% COCO data setup. Furthermore, this controllability enables targeted data generation for intra-class referring, a diagnostic grounding task we propose that requires fine-grained attribute discrimination.

Figures (15)

• Figure 1: SOC data examples. With SOC, we generate 20M object segments and compose them into 2M images via 3D geometric layout augmentation, producing accurate and diverse masks, bounding boxes, and referring expressions for object detection, instance segmentation, and visual grounding. Right: SOC surpasses both real and synthetic data generation pipelines and complements real datasets when combined.
• Figure 2: Overview of SOC Pipeline. (1) Object Segments Generation. Generating object segments in diverse categories. (2) 3D Geometric Layout Augmentation. Sample and place 5–20 segments per image with category-independent 3D scene modeling. (3) Generative Harmonization. Apply inpainting, global relighting, and mask-area-weighted blending to enhance realism and prevent models from learning only the edge without semantics. (4) Camera Configuration Augmentation. Apply random scaling, cropping, and depth-of-field blur to simulate diverse camera configurations. (5) Generating Region Annotations. Compute final masks, bounding boxes, and dense referring expressions.
• Figure 3: Comparison of Generative Harmonization Strategies. From left to right: (1) input foreground images, (2) naively pasted onto a background, (3) Inpainting and Inpainting and global relighting with IC-Light (AP = 36.3 on LVIS-Mini), and (4) Inpainting plus relintiingng plus relighting plus mask-area-weighted blending (AP = 38.6 (+2.3) on LVIS-Mini). Blending notably preserves fine details of small objects and color fidelity and improves model performance.
• Figure 4: (Sec. \ref{['exp:ovs']}) SOS consistently improves LVIS instance segmentation, with the largest gains on rare categories.
• Figure 5: (Sec. \ref{['exp:mask2former_lowdata']}) Combining SOC synthetic segments with real COCO segments increases AP over all COCO data scales.