AnchorDS: Anchoring Dynamic Sources for Semantically Consistent Text-to-3D Generation

TL;DR

AnchorDS reframes text-to-3D optimization as a dynamic editing process by acknowledging that the source distribution evolves with the 3D state. It introduces a dual-conditioned diffusion framework that anchors the current render with an image condition while guiding toward the text-conditioned target, using a pseudo-source reconstruction and lightweight filtering/finetuning to stabilize guidance. The method consistently outperforms vanilla SDS, SDS-Bridge, and VSD in 3D consistency, semantic fidelity, and efficiency across Gaussian Splatting and NeRF pipelines, using both IP-Adapter and ControlNet as image conditioners. By leveraging the intermediate rendering as a self-referential anchor, AnchorDS achieves finer detail and stronger semantic alignment for complex prompts, enabling robust and scalable text-to-3D generation.

Abstract

Optimization-based text-to-3D methods distill guidance from 2D generative models via Score Distillation Sampling (SDS), but implicitly treat this guidance as static. This work shows that ignoring source dynamics yields inconsistent trajectories that suppress or merge semantic cues, leading to "semantic over-smoothing" artifacts. As such, we reformulate text-to-3D optimization as mapping a dynamically evolving source distribution to a fixed target distribution. We cast the problem into a dual-conditioned latent space, conditioned on both the text prompt and the intermediately rendered image. Given this joint setup, we observe that the image condition naturally anchors the current source distribution. Building on this insight, we introduce AnchorDS, an improved score distillation mechanism that provides state-anchored guidance with image conditions and stabilizes generation. We further penalize erroneous source estimates and design a lightweight filter strategy and fine-tuning strategy that refines the anchor with negligible overhead. AnchorDS produces finer-grained detail, more natural colours, and stronger semantic consistency, particularly for complex prompts, while maintaining efficiency. Extensive experiments show that our method surpasses previous methods in both quality and efficiency.

Figures (13)

• Figure 1: Comparison of Vanilla SDS vs. Ours. SDS suffers from semantic over-smoothing (mixing swan and lake) and cross-view inconsistency (multiple heads). Ours achieves semantic disentanglement and 3D consistency across views.
• Figure 2: Static vs. Dynamic Pseudo-Editing. Static source estimation leverages an unconditional prior, misaligned with the actual 3D state. Instead, the dynamic pseudo-source reflects the evolving 3D renderings, ensuring faithful guidance.
• Figure 3: Overview of Our AnchorDS for Text-to-3D Generation.(a) Optimization-based 3D generation as an evolving editing process where each step refines the 3D model guided by AnchorDS. (b) Each editing step bridges the source distribution to a target distribution through dynamic anchoring in latent space. (c) Technical framework showing how $L_{AnchorDS}$ and $L_{rec}$ enable dynamic source anchoring, with source and target branches processing the evolving 3D content through pretrained diffusion models.
• Figure 4: Source Estimation Strategies. (a) Vanilla SDS poorly reconstructs the source. (b) AnchorDS, conditioned on the current rendering, recovers geometry and color faithfully. (c) Our fine-tuning strategy achieves an accurate match to the source image.
• Figure 5: Qualitative Comparison to Existing Methods on SD 1.5 on complex text prompts, including (1) objects with rich or mixed semantics; (2) multiple distinct objects; (3) objects with fine-grained details. SDS-Bridge produces biased material textures and inaccurate semantics, likely due to new biases introduced by the handcrafted prompt. VSD fails to recover coherent structures and exhibits exaggerated, unrealistic colours. In contrast, our method consistently generates semantically faithful and structurally accurate results.