RecDreamer: Consistent Text-to-3D Generation via Uniform Score Distillation

TL;DR

RecDreamer tackles pose-induced geometric inconsistencies in text-to-3D diffusion generation by rectifying the underlying data distribution to yield a uniform pose marginal. It introduces a rectification function that modulates the data density and defines Uniform Score Distillation (USD) to align the 3D generator with the rectified distribution, while preserving the diffusion sampling process. A training-free pose classifier estimates pose categories to compute the rectification term, and a set of approximations handles noisy states and real-time distribution updates. Empirical results show improved pose consistency and fewer Multi-Face Janus artifacts across multiple views with comparable rendering quality, demonstrating the method's practical impact for more reliable, view-consistent 3D synthesis. The framework also offers cross-domain applicability and a pathway for addressing other biases in diffusion priors.

Abstract

Current text-to-3D generation methods based on score distillation often suffer from geometric inconsistencies, leading to repeated patterns across different poses of 3D assets. This issue, known as the Multi-Face Janus problem, arises because existing methods struggle to maintain consistency across varying poses and are biased toward a canonical pose. While recent work has improved pose control and approximation, these efforts are still limited by this inherent bias, which skews the guidance during generation. To address this, we propose a solution called RecDreamer, which reshapes the underlying data distribution to achieve a more consistent pose representation. The core idea behind our method is to rectify the prior distribution, ensuring that pose variation is uniformly distributed rather than biased toward a canonical form. By modifying the prescribed distribution through an auxiliary function, we can reconstruct the density of the distribution to ensure compliance with specific marginal constraints. In particular, we ensure that the marginal distribution of poses follows a uniform distribution, thereby eliminating the biases introduced by the prior knowledge. We incorporate this rectified data distribution into existing score distillation algorithms, a process we refer to as uniform score distillation. To efficiently compute the posterior distribution required for the auxiliary function, RecDreamer introduces a training-free classifier that estimates pose categories in a plug-and-play manner. Additionally, we utilize various approximation techniques for noisy states, significantly improving system performance. Our experimental results demonstrate that RecDreamer effectively mitigates the Multi-Face Janus problem, leading to more consistent 3D asset generation across different poses.

Figures (22)

• Figure 1: The Multi-Face Janus problem arises from an imbalance in the pose distribution of pretrained models, which tend to generate predominantly frontal images. This bias results in excessive faces appearing in the generated 3D assets. RecDreamer addresses this issue by producing a distribution with a uniform pose marginal, enabling more diverse pose generation and mitigating the Multi-Face Janus problem.
• Figure 2: The architecture of our classifier combines orientation and texture similarities in a differential "and-gate" manner. Orientation similarity is evaluated using a patch-matching distance metric, while texture similarity is calculated via cosine similarity of the $[cls]$ token.
• Figure 3: Qualitative comparison. The text-to-3D generation results are visualized from three perspectives (front, left, and right side views), illustrating how the rectified distribution in our USD framework effectively mitigates the Multi-Face Janus phenomenon.
• Figure 4: Ablation studies. We construct two variants, $p(c|\boldsymbol{x}_t, y)$ and $p_t(c|y)$ for comparison. The variant $p(c|\boldsymbol{x}_t, y)$ directly predicts the category of noisy images $\boldsymbol{x}_t$, while the sampling-based method, $p_t(c|y)$, estimates the pose distribution by generating multiple samples and predicting the respective categories.
• Figure 5: Calculation of orientation distance. (a) is the input image for testing. (e) and (j) are templates showing "facing right" and "facing left" orientations. Each row demonstrates the orientation distance between the specific image patch (d) and the templates. For opposite orientations, the mean distance is larger compared to the similar orientations ((i) v.s. (n)). This feature is important for the classification of pose.

Theorems & Definitions (11)

• Theorem 1: Proof in Appendix \ref{['app:method_theorem']}
• Theorem 2: Proof in Appendix \ref{['app:method_theorem']}
• Corollary 1: Corollary to Theorem 2 from VSD
• Lemma 1
• proof : Proof of Lemma \ref{['lm:weight']}
• proof : Proof of Theorem \ref{['thm:rpx']}
• Corollary 2
• proof : Proof of Corollary \ref{['cr:rpx_cond']}
• Remark 1
• proof : Proof of Theorem \ref{['thm:rpx0t_cond']}