Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Score Distilling Sampling for 3D Editing and Generation

Xingyu Miao, Haoran Duan, Yang Long, Jungong Han

TL;DR

This work addresses the lack of a unified approach to text-guided 3D editing and generation under SDS-based diffusion priors. By analyzing DDS and PDS through their gradient terms and connecting them to DDIM-based variants, the authors propose Unified Distillation Sampling (UDS), which integrates identity-preserving reconstruction with classifier-free guidance in a single gradient formulation and leverages DDIM inversion to estimate latent $x_0$. Across NeRF and 3D Gaussian Splatting, UDS delivers superior editing and generation results, with lower gradient variability, better detail preservation, and stronger identity retention at practical classifier-free guidance weights. The method demonstrates improved performance on 3D editing and generation tasks, supported by quantitative metrics (e.g., CLIP) and human studies, and offers a more interpretable, unified framework for 3D content creation.

Abstract

Score Distillation Sampling (SDS) has emerged as a prominent method for text-to-3D generation by leveraging the strengths of 2D diffusion models. However, SDS is limited to generation tasks and lacks the capability to edit existing 3D assets. Conversely, variants of SDS that introduce editing capabilities often can not generate new 3D assets effectively. In this work, we observe that the processes of generation and editing within SDS and its variants have unified underlying gradient terms. Building on this insight, we propose Unified Distillation Sampling (UDS), a method that seamlessly integrates both the generation and editing of 3D assets. Essentially, UDS refines the gradient terms used in vanilla SDS methods, unifying them to support both tasks. Extensive experiments demonstrate that UDS not only outperforms baseline methods in generating 3D assets with richer details but also excels in editing tasks, thereby bridging the gap between 3D generation and editing. The code is available on: https://github.com/xingy038/UDS.

Rethinking Score Distilling Sampling for 3D Editing and Generation

TL;DR

This work addresses the lack of a unified approach to text-guided 3D editing and generation under SDS-based diffusion priors. By analyzing DDS and PDS through their gradient terms and connecting them to DDIM-based variants, the authors propose Unified Distillation Sampling (UDS), which integrates identity-preserving reconstruction with classifier-free guidance in a single gradient formulation and leverages DDIM inversion to estimate latent . Across NeRF and 3D Gaussian Splatting, UDS delivers superior editing and generation results, with lower gradient variability, better detail preservation, and stronger identity retention at practical classifier-free guidance weights. The method demonstrates improved performance on 3D editing and generation tasks, supported by quantitative metrics (e.g., CLIP) and human studies, and offers a more interpretable, unified framework for 3D content creation.

Abstract

Score Distillation Sampling (SDS) has emerged as a prominent method for text-to-3D generation by leveraging the strengths of 2D diffusion models. However, SDS is limited to generation tasks and lacks the capability to edit existing 3D assets. Conversely, variants of SDS that introduce editing capabilities often can not generate new 3D assets effectively. In this work, we observe that the processes of generation and editing within SDS and its variants have unified underlying gradient terms. Building on this insight, we propose Unified Distillation Sampling (UDS), a method that seamlessly integrates both the generation and editing of 3D assets. Essentially, UDS refines the gradient terms used in vanilla SDS methods, unifying them to support both tasks. Extensive experiments demonstrate that UDS not only outperforms baseline methods in generating 3D assets with richer details but also excels in editing tasks, thereby bridging the gap between 3D generation and editing. The code is available on: https://github.com/xingy038/UDS.
Paper Structure (28 sections, 28 equations, 15 figures, 3 tables, 1 algorithm)

This paper contains 28 sections, 28 equations, 15 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Methodology
  5. Revisiting SDS Variant for Editing
  6. Unified Distillation Sampling (UDS)
  7. Discuss with PDS and ISM
  8. Experiments
  9. Experiment Setup
  10. Text Guided 3D Editing
  11. Text-to-3D Generation
  12. Conclusion
  13. Appendix
  14. Connection with other generation methods
  15. Workflows
  16. ...and 13 more sections

Figures (15)

  • Figure 1: Example of text-guided 3D editing and text-to-3D content generated from scratch by our UDS. We achieve superior 3D editing and 3D generation results with photorealistic quality in a short training time. Please zoom in for details.
  • Figure 2: Comparison with baseline methods in text-guided 3D editing. We present visual editing results for both our UDS and the baseline methods. Experiments show that our UDS effectively edits 3D content to closely align with the input text prompts, while maintaining a high level of photorealism. Notably, DDS hertz2023delta and PDS koo2024posterior set CFG is 100, while our is 7.5.
  • Figure 3: Comparison with baseline methods in text-to-3D generation. Experiments demonstrate that our approach can generate 3D content that closely aligns with the input text prompts, exhibiting high fidelity and intricate details. The running time of all methods is measured on a single 3090 GPU. Notably, we tried to reproduce ProlificDreamer and LucidDreamer, but failed to achieve the results in the LucidDreamer paper. Therefore, we directly use the visualization results in the LucidDreamer paper for these two methods.
  • Figure 4: Ablation for different approximate methods.
  • Figure 5: Ablation for SDS poole2022dreamfusion and UDS with different generation frameworks.
  • ...and 10 more figures