VoxHammer: Training-Free Precise and Coherent 3D Editing in Native 3D Space

TL;DR

VoxHammer presents a training-free framework for precise and coherent 3D local editing in native 3D space by performing inversion in a pretrained 3D diffusion model (TRELLIS) and applying latent and key-value feature replacements during denoising. A two-stage inversion (ST and SLAT) with cached K/V tokens enables faithful reconstruction of unedited regions, while mask-guided edits ensure seamless integration of changes. The authors introduce Edit3D-Bench for objective 3D editing evaluation and demonstrate superior performance over baselines in both local preservation and overall 3D quality, with additional demonstrations across part-aware, scene, and NeRF/3DGS editing. The work further shows potential for generating high-quality edited paired data to support in-context 3D generation, albeit with limitations in text alignment fidelity and processing time.

Abstract

3D local editing of specified regions is crucial for game industry and robot interaction. Recent methods typically edit rendered multi-view images and then reconstruct 3D models, but they face challenges in precisely preserving unedited regions and overall coherence. Inspired by structured 3D generative models, we propose VoxHammer, a novel training-free approach that performs precise and coherent editing in 3D latent space. Given a 3D model, VoxHammer first predicts its inversion trajectory and obtains its inverted latents and key-value tokens at each timestep. Subsequently, in the denoising and editing phase, we replace the denoising features of preserved regions with the corresponding inverted latents and cached key-value tokens. By retaining these contextual features, this approach ensures consistent reconstruction of preserved areas and coherent integration of edited parts. To evaluate the consistency of preserved regions, we constructed Edit3D-Bench, a human-annotated dataset comprising hundreds of samples, each with carefully labeled 3D editing regions. Experiments demonstrate that VoxHammer significantly outperforms existing methods in terms of both 3D consistency of preserved regions and overall quality. Our method holds promise for synthesizing high-quality edited paired data, thereby laying the data foundation for in-context 3D generation. See our project page at https://huanngzh.github.io/VoxHammer-Page/.

Figures (10)

• Figure 1: High-quality 3D assets edited by our method using text prompts. Our method uses a training-free approach to perform percise and coherent 3D local editing, transforming multiple 3D assets in the scene (left) into high-quality results (right). The bottom row shows a detailed comparison of each 3D asset before and after editing, as well as the conditioning texts.
• Figure 2: Pipeline. Given an input 3D model, a user-specified editing region, and a text prompt, the off-the-shelf models flux2024fluxfill are used to inpaint the rendered view from the 3D model. Subsequently, our VoxHammer, a training-free framework based on structured 3D diffusion models trellis, performs native 3D editing conditioned on the input 3D and the edited image.
• Figure 3: Architecture of VoxHammer. Our framework adopts TRELLIS trellis as the base model, which predicts sparse structures at the first structure (ST) stage and denoise fine-grained structured latents at the second sparse-latent (SLAT) stage. VoxHammer performs inversion prediction in both the ST and SLAT stages, which map the textured 3D asset to its terminal noise, with latents and key/value tensors cached at each timestep. Subsequently, VoxHammer denoises from the inverted noise, and replace the features of the preserved regions with the corresponding cached latents and key-value tokens, thereby achieving precise and coherent editing in native 3D space.
• Figure 4: Qualitative comparisons on Edit3D-Bench. Our method achieves best performance on precision of editing and overall quality.
• Figure 5: Ablation studies. Results demonstrate the effectiveness of key-value replacement in attention mechanism and latent replacement.