Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference

TL;DR

This work fundamentally rethink preference alignment as a classifier-free guidance (CFG)-style mechanism through the authors' implicit reward model, and is the first to bridge human preference alignment with CFG theory under score distillation framework.

Abstract

Human preference alignment presents a critical yet underexplored challenge for diffusion models in text-to-3D generation. Existing solutions typically require task-specific fine-tuning, posing significant hurdles in data-scarce 3D domains. To address this, we propose Preference Score Distillation (PSD), an optimization-based framework that leverages pretrained 2D reward models for human-aligned text-to-3D synthesis without 3D training data. Our key insight stems from the incompatibility of pixel-level gradients: due to the absence of noisy samples during reward model training, direct application of 2D reward gradients disturbs the denoising process. Noticing that similar issue occurs in the naive classifier guidance in conditioned diffusion models, we fundamentally rethink preference alignment as a classifier-free guidance (CFG)-style mechanism through our implicit reward model. Furthermore, recognizing that frozen pretrained diffusion models constrain performance, we introduce an adaptive strategy to co-optimize preference scores and negative text embeddings. By incorporating CFG during optimization, online refinement of negative text embeddings dynamically enhances alignment. To our knowledge, we are the first to bridge human preference alignment with CFG theory under score distillation framework. Experiments demonstrate the superiority of PSD in aesthetic metrics, seamless integration with diverse pipelines, and strong extensibility.

Figures (19)

• Figure 1: Comparisons with state-of-the-art methods RichDreamer richdreamer and Trellis trellis. Even when compared to methods that leverage stronger 3D priors, our method achieves significantly higher text alignment and enhanced visual quality, highlighting the critical role of preference alignment in text-to-3D generation.
• Figure 2: Overall illustration of Preference Score Distillation. a) Win (red) and lose (blue) samples are constructed on-the-fly to calculate win and lose scores, then preference score guidance (purple) pushes the denoising trajectory towards high-reward regions and finally improve alignment with reward. b) In each step, two noise is added to the rendering images $g_{\theta}(\boldsymbol{c})$ and reward model determines win/lose based on one-step prediction of pretrained diffusion models $\boldsymbol{\epsilon}_\phi$. 3D representation $\theta$ and negative embedding $n$ are updated by our objective $\mathcal{L}_{\text{PSD}}(\theta)$ and reward score respectively.
• Figure 3: Effect of negative embedding optimization strategy on single prompt. Employ negative embedding optimization can significantly improve aesthetic score, but overlarge learning rate will harm visual quality.
• Figure 4: Qualitative comparison of single-stage distillation of MVDream mvdream ($256\times256$). Our PSD significantly improve text alignment (red) and visual quality against comparing method DreamReward dreamreward and DreamDPO dreamdpo.
• Figure 5: Qualitative comparison of 2-stage NeRF nerf ($512\times512$) and 3-Stage DMTet dmtet ($1024\times1024$) generation. PSD improves alignment with the prompts in red.

Theorems & Definitions (2)

• Proposition 1
• proof