PlaceIt3D: Language-Guided Object Placement in Real 3D Scenes

TL;DR

This work defines PlaceIt3D, the task of language-guided object placement in real 3D scenes, where a 3D asset must be positioned and oriented to satisfy a textual prompt while obeying physical constraints. It introduces PlaceIt3D-benchmark for evaluating one-to-many valid placements and PlaceIt3D-dataset for training, along with PlaceWizard, a proto-baseline that fuses 3D scene and asset encodings with a language-conditioned decoder to predict placement masks. Through extensive ablations and comparisons to baselines, PlaceWizard demonstrates strong adherence to language constraints and plausible placements, while highlighting remaining challenges in physical accuracy and instruction consistency. The work provides a foundation for integrating natural language guidance with 3D spatial reasoning in real-world scenes, with implications for AR, robotics, and interactive design tooling.

Abstract

We introduce the novel task of Language-Guided Object Placement in Real 3D Scenes. Our model is given a 3D scene's point cloud, a 3D asset, and a textual prompt broadly describing where the 3D asset should be placed. The task here is to find a valid placement for the 3D asset that respects the prompt. Compared with other language-guided localization tasks in 3D scenes such as grounding, this task has specific challenges: it is ambiguous because it has multiple valid solutions, and it requires reasoning about 3D geometric relationships and free space. We inaugurate this task by proposing a new benchmark and evaluation protocol. We also introduce a new dataset for training 3D LLMs on this task, as well as the first method to serve as a non-trivial baseline. We believe that this challenging task and our new benchmark could become part of the suite of benchmarks used to evaluate and compare generalist 3D LLM models.

Figures (8)

• Figure 1: Language-guided 3D Object Placement in Real 3D Scenes: Given a text prompt, the task is to find a valid placement for an asset, requiring semantic and geometric understanding of the scene, the asset’s shape, and spatial relationships. Colored dots show referenced objects (for visualization only, not given to the model), and the yellow arrow indicates the predicted frontal direction.
• Figure 2: PlaceIt3D-dataset creation. Given a scene and an asset as input (a) the goal is to create a prompt (f) and corresponding mask $\mathcal{M}$ of valid placements (e). We start by finding the set of points which are physically plausible placements, shown in red in (b). We consider eight equally spaced rotation angles, which condition the valid placements. For this example, angle $0^{\circ}$ has more valid placements than $45^{\circ}$. To generate the language constraints, we use the ground truth scene graph (c). Object anchors are selected from the scene graph and combined with relationship types to create a constraint and corresponding validity mask (d). The different placement constraints are combined in the final output by intersecting the validity masks (e) given a mask of valid dense placements. Based on each selected set of anchors and constraint relationships, a natural language prompt is created using templates (please, see supplemental for more details).
• Figure 3: PlaceWizard overview. A point encoder extracts features from the 3D scene, which are then complemented with positional embeddings. Spatial pooling reduces feature dimensions, and a Q-Former merges the pooled features with trainable queries $Q$ (Section \ref{['sec:scene_encoding']}). The asset is encoded into a single vector by using a pretrained asset encoder followed by max-pooling (Section \ref{['sec:asset_encoding']}). This vector together with a size embedding is passed to a projection layer that aligns the features with the LLM space. The LLM take as input (i) the output of the Q-Former, (ii) the text prompt, and (iii) the projected asset features and predicts three special tokens $\mathrm{[ANC]}$, $\mathrm{[LOC]}$ and $\mathrm{[ROT]}$. A transformer based decoder takes as input the features associated with the three special tokens and the pooled scene features and performs a few self and cross attention operations (Section \ref{['sec:decoder']}). Three heads produce the final outputs: $\mathcal{M}_{loc}$ the valid placement mask; $\mathcal{M}_{anc}$ an auxiliary mask that localizes the object anchors; and $\mathcal{M}_{rot}$ a mask indicating which rotation angles are valid at each location.
• Figure 4: Qualitative benchmark results. Colored highlights indicate anchors referenced in the textual prompts (predictions are generated entirely from point clouds, with anchor information provided only as text). The asset position is marked with a yellow circle, and a yellow arrow denotes the frontal orientation. Our method successfully follows language instructions and meets the specified constraints. The top-right example illustrates a placement that satisfies constraints but slightly intersects with the scene mesh. The bottom-right example demonstrates a failure case where one constraint is not met (highlighted in red).
• Figure 5: Comparison of our spatial pooling vs the superpoints used in huang2024reason3d. Our regions are more local and more adequate to the task of object placement.