FreeInsert: Disentangled Text-Guided Object Insertion in 3D Gaussian Scene without Spatial Priors

TL;DR

FreeInsert tackles unsupervised text-guided object insertion in 3D Gaussian scenes without spatial priors. It decouples object generation from spatial placement and leverages foundation models to extract structured semantics, initialize 3D degrees of freedom, and guide hierarchical refinement and appearance enhancement. The approach delivers semantically coherent, spatially precise insertions with high visual fidelity, demonstrated across diverse scenes and object types and showing competitive quantitative performance against priors-based baselines. By eliminating manual priors, FreeInsert broadens practical 3D scene editing and enables flexible, language-driven modifications in open-world settings.

Abstract

Text-driven object insertion in 3D scenes is an emerging task that enables intuitive scene editing through natural language. However, existing 2D editing-based methods often rely on spatial priors such as 2D masks or 3D bounding boxes, and they struggle to ensure consistency of the inserted object. These limitations hinder flexibility and scalability in real-world applications. In this paper, we propose FreeInsert, a novel framework that leverages foundation models including MLLMs, LGMs, and diffusion models to disentangle object generation from spatial placement. This enables unsupervised and flexible object insertion in 3D scenes without spatial priors. FreeInsert starts with an MLLM-based parser that extracts structured semantics, including object types, spatial relationships, and attachment regions, from user instructions. These semantics guide both the reconstruction of the inserted object for 3D consistency and the learning of its degrees of freedom. We leverage the spatial reasoning capabilities of MLLMs to initialize object pose and scale. A hierarchical, spatially aware refinement stage further integrates spatial semantics and MLLM-inferred priors to enhance placement. Finally, the appearance of the object is improved using the inserted-object image to enhance visual fidelity. Experimental results demonstrate that FreeInsert achieves semantically coherent, spatially precise, and visually realistic 3D insertions without relying on spatial priors, offering a user-friendly and flexible editing experience.

Figures (9)

• Figure 1: "No Spatial Priors, Just Prompts." Compared to existing methods that require user-provided spatial priors, limiting their practicality, our method enables flexible text-driven object insertion without any need for such priors (e.g., 2D masks or 3D bounding boxes). Given only a text prompt (The image prompt is optional), FreeInsert naturally inserts objects across diverse scenes.
• Figure 2: Overview of FreeInsert. Given an text prompt $\mathcal{T}$ and optionally an image prompt $\mathcal{I}_{\textit{O}}$, the object insertion process includes four stages: (a) The MLLM-based Object Insertion Parser (see \ref{['sec:llm_parser']}) first extracts structured semantics to support the subsequent stages. (b) The Initialization via Large Models (see \ref{['sec:dof_init']}) stage generates object and initializes its $\textit{DoF}_{\textit{O}}$ in the scene . (c) The Hierarchical Spatial Aware Refinement (see \ref{['sec:spatial_aware']}) stage refines the $\textit{DoF}_{\textit{O}}$. (d) The final stage, Object Appearance Refinement (see \ref{['sec:appear_opt']}), enhances the object’s visual quality using object image$\mathcal{I}_{\textit{O}}$.
• Figure 3: Visual comparison with state-of-the-art methods for text-guided object insertion (Cols 1–2) and replacement (Col 3). Our method generates higher-quality results while preserving scene integrity. IN2N (GS) and GaussCtrl sometimes misunderstand the prompt and fail to complete insertion (e.g., “Give the doll a pair of glasses to the doll”), and struggle to produce clear shape changes in replacement (Col 3, Rows 2–3). GaussianEditor requires manual masks and depth adjustment, and suffers from artifacts and low-quality objects due to post-inpainting and 3D reconstruction limitations.
• Figure 4: Visual comparisons with TIP-Editor using text-image prompts. Our method achieves competitive results with TIP-Editor, without relying on 3D bounding boxes. TIP-Editor struggles to maintain the 3D consistency of the inserted object (e.g., the misaligned hat across views in row 2, column 2, and the right front paw intersecting with the left in row 5, column 2), as its 2D editing process lacks cross-view constraints. Our method produces clearly more 3D-consistent results and more closely resembles the reference image.
• Figure 5: Visualization of different stages in FreeInsert