SCENEFORGE: Enhancing 3D-text alignment with Structured Scene Compositions

TL;DR

We address the scarcity of large-scale 3D–text data by introducing SceneForge, a compositional data framework that builds multi-object 3D scenes with explicit spatial relations and refines their captions with a large language model. SceneForge integrates two modules—3D Scene Forge and Scene Caption Forge—to generate structured samples that are mixed with single-object examples in training, and applied within a batch-level contrastive learning objective that preserves CLIP-like alignment while focusing on 3D–text pairs. Across zero-shot classification, few-shot segmentation, 3D VQA, and retrieval benchmarks, SceneForge delivers consistent gains across multiple backbones, demonstrating encoder-agnostic benefits and improved spatial relational reasoning. The approach expands data diversity and enhances generalization, with practical impact for 3D scene understanding in robotics, AR/VR, and related fields, while identifying limitations such as a limited relation set and caption-noise at high composition counts.

Abstract

The whole is greater than the sum of its parts-even in 3D-text contrastive learning. We introduce SceneForge, a novel framework that enhances contrastive alignment between 3D point clouds and text through structured multi-object scene compositions. SceneForge leverages individual 3D shapes to construct multi-object scenes with explicit spatial relations, pairing them with coherent multi-object descriptions refined by a large language model. By augmenting contrastive training with these structured, compositional samples, SceneForge effectively addresses the scarcity of large-scale 3D-text datasets, significantly enriching data complexity and diversity. We systematically investigate critical design elements, such as the optimal number of objects per scene, the proportion of compositional samples in training batches, and scene construction strategies. Extensive experiments demonstrate that SceneForge delivers substantial performance gains across multiple tasks, including zero-shot classification on ModelNet, ScanObjNN, Objaverse-LVIS, and ScanNet, as well as few-shot part segmentation on ShapeNetPart. SceneForge's compositional augmentations are model-agnostic, consistently improving performance across multiple encoder architectures. Moreover, SceneForge improves 3D visual question answering on ScanQA, generalizes robustly to retrieval scenarios with increasing scene complexity, and showcases spatial reasoning capabilities by adapting spatial configurations to align precisely with textual instructions.

Figures (7)

• Figure 1: Our multimodal scene composition framework for contrastive 3D-text learning. Given a batch of point clouds and their captions, each sample is randomly kept as single object or combined into a synthetic 3D scene with a random number of objects. Objects designated for combination are passed to the SceneForge module, which samples the additional objects and their captions. Spatial relationships among the selected objects are randomly assigned, and the 3D Scene Forge and Scene Caption Forge generate the corresponding combined 3D scene and its composite caption. The newly formed scenes are merged with the unmodified objects to construct a final batch, which is then used for contrastive 3D-text alignment using a pretrained CLIP text encoder.
• Figure 2: Scene Caption Forge. Starting from the initial caption ($t_0$), each caption ($t_i$) is connected using its relative position ($\text{s}_i$), creating a raw combined caption $c^{\text{txt,raw}}$. The raw caption $c^{\text{txt,raw}}$ is then refined to the final $c^{\text{txt}}$ using Qwen2.5.
• Figure 3: Top-1 accuracy across different datasets (Lvis, ModelNet, ScanObjectNN, and Scannet) as a function of the number of combined objects.
• Figure 4: Top‑1 averaged retrieval accuracy on the N‑LVIS datasets as $N$ increases.
• Figure 5: Object repositioning example.