Error-Driven Scene Editing for 3D Grounding in Large Language Models

TL;DR

This work tackles grounding in 3D-language models by addressing biases arising from limited 3D data. It introduces DEER-3D, an error-driven framework that decomposes instructions, diagnoses predicate-level grounding failures, applies controlled Clone–Replace–Modify edits to 3D scenes, and augments training with aligned QA in a closed-loop retraining process. Across standard 3D grounding benchmarks, DEER-3D yields consistent improvements, with additional gains when using multimodal inputs and through iterative refinement that progressively reduces grounding errors. The results demonstrate that predicate-level, counterfactual visual editing in 3D space can significantly enhance the alignment between linguistic reasoning and spatial grounding in 3D-LLMs.

Abstract

Despite recent progress in 3D-LLMs, they remain limited in accurately grounding language to visual and spatial elements in 3D environments. This limitation stems in part from training data that focuses on language reasoning rather than spatial understanding due to scarce 3D resources, leaving inherent grounding biases unresolved. To address this, we propose 3D scene editing as a key mechanism to generate precise visual counterfactuals that mitigate these biases through fine-grained spatial manipulation, without requiring costly scene reconstruction or large-scale 3D data collection. Furthermore, to make these edits targeted and directly address the specific weaknesses of the model, we introduce DEER-3D, an error-driven framework following a structured "Decompose, Diagnostic Evaluation, Edit, and Re-train" workflow, rather than broadly or randomly augmenting data as in conventional approaches. Specifically, upon identifying a grounding failure of the 3D-LLM, our framework first diagnoses the exact predicate-level error (e.g., attribute or spatial relation). It then executes minimal, predicate-aligned 3D scene edits, such as recoloring or repositioning, to produce targeted counterfactual supervision for iterative model fine-tuning, significantly enhancing grounding accuracy. We evaluate our editing pipeline across multiple benchmarks for 3D grounding and scene understanding tasks, consistently demonstrating improvements across all evaluated datasets through iterative refinement. DEER-3D underscores the effectiveness of targeted, error-driven scene editing in bridging linguistic reasoning capabilities with spatial grounding in 3D LLMs.

Figures (6)

• Figure 1: 3D-LLMs frequently overfit to dataset co-occurrence biases (e.g., white pillows), causing grounding failures. We mitigate these biases through targeted counterfactual edits in 3D scenes and construct aligned QA pairs to strengthen the model’s grounding ability.
• Figure 2: Overview of DEER-3D. When a grounding error is detected, DEER-3D performs targeted visual edits. Given a natural-language instruction, the framework (a) decomposes it into atomic predicates, (b) diagnoses the specific error, and (c) applies predicate-level visual edits. Aligned question–answer pairs are then created to explicitly supervise the failed predicate. Finally, the model (d) iteratively retrains on these counterfactual examples to progressively improve grounding accuracy.
• Figure 3: Examples of targeted visual edits for different grounding errors. Each edit generates aligned question–answer pairs, precisely supervising the model's visual grounding ability.
• Figure 4: Ablation studies validating our method. (a) Grounding performance scales with the quantity of our counterfactual data. (b) Our iterative loop (Round 2 vs. Round 1) successfully reduces all targeted error types. (c) Our full Err-Guided-Mix strategy is superior to both Random-aug and individual components, validating our error-driven and complementary design.
• Figure 5: Distribution of semantic predicate types from our instruction analysis.