Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text-to-Image Rectified Flow as Plug-and-Play Priors

Xiaofeng Yang, Cheng Chen, Xulei Yang, Fayao Liu, Guosheng Lin

TL;DR

This work introduces rectified-flow priors as plug-and-play priors for generative tasks, developing RFDS, iRFDS, and RFDS-Rev to distill knowledge from pretrained rectified-flow models. RFDS provides a baseline distillation aligned with SDS-like objectives, while iRFDS enables image inversion/editing by exploiting time-symmetry, and RFDS-Rev further boosts quality via a two-stage reversal mechanism. Across 2D and 3D tasks, especially text-to-3D generation, RFDS and RFDS-Rev outperform diffusion priors like SDS and VSD, achieving state-of-the-art results on 2D lifting benchmarks and competitive or superior performance for 3D lifting with faster convergence. The methods extend to diffusion models via PF-ODE formulations and CFG variants, offering practical benefits in speed and editing capabilities, though limitations remain in camera pose handling and CFG-mismatch scenarios for 2D-to-3D translation. Overall, the paper broadens the utility of rectified flow models as priors, delivering effective, efficient alternatives to diffusion priors for text-to-3D generation, inversion, and editing with strong empirical support.

Abstract

Large-scale diffusion models have achieved remarkable performance in generative tasks. Beyond their initial training applications, these models have proven their ability to function as versatile plug-and-play priors. For instance, 2D diffusion models can serve as loss functions to optimize 3D implicit models. Rectified flow, a novel class of generative models, enforces a linear progression from the source to the target distribution and has demonstrated superior performance across various domains. Compared to diffusion-based methods, rectified flow approaches surpass in terms of generation quality and efficiency, requiring fewer inference steps. In this work, we present theoretical and experimental evidence demonstrating that rectified flow based methods offer similar functionalities to diffusion models - they can also serve as effective priors. Besides the generative capabilities of diffusion priors, motivated by the unique time-symmetry properties of rectified flow models, a variant of our method can additionally perform image inversion. Experimentally, our rectified flow-based priors outperform their diffusion counterparts - the SDS and VSD losses - in text-to-3D generation. Our method also displays competitive performance in image inversion and editing.

Text-to-Image Rectified Flow as Plug-and-Play Priors

TL;DR

This work introduces rectified-flow priors as plug-and-play priors for generative tasks, developing RFDS, iRFDS, and RFDS-Rev to distill knowledge from pretrained rectified-flow models. RFDS provides a baseline distillation aligned with SDS-like objectives, while iRFDS enables image inversion/editing by exploiting time-symmetry, and RFDS-Rev further boosts quality via a two-stage reversal mechanism. Across 2D and 3D tasks, especially text-to-3D generation, RFDS and RFDS-Rev outperform diffusion priors like SDS and VSD, achieving state-of-the-art results on 2D lifting benchmarks and competitive or superior performance for 3D lifting with faster convergence. The methods extend to diffusion models via PF-ODE formulations and CFG variants, offering practical benefits in speed and editing capabilities, though limitations remain in camera pose handling and CFG-mismatch scenarios for 2D-to-3D translation. Overall, the paper broadens the utility of rectified flow models as priors, delivering effective, efficient alternatives to diffusion priors for text-to-3D generation, inversion, and editing with strong empirical support.

Abstract

Large-scale diffusion models have achieved remarkable performance in generative tasks. Beyond their initial training applications, these models have proven their ability to function as versatile plug-and-play priors. For instance, 2D diffusion models can serve as loss functions to optimize 3D implicit models. Rectified flow, a novel class of generative models, enforces a linear progression from the source to the target distribution and has demonstrated superior performance across various domains. Compared to diffusion-based methods, rectified flow approaches surpass in terms of generation quality and efficiency, requiring fewer inference steps. In this work, we present theoretical and experimental evidence demonstrating that rectified flow based methods offer similar functionalities to diffusion models - they can also serve as effective priors. Besides the generative capabilities of diffusion priors, motivated by the unique time-symmetry properties of rectified flow models, a variant of our method can additionally perform image inversion. Experimentally, our rectified flow-based priors outperform their diffusion counterparts - the SDS and VSD losses - in text-to-3D generation. Our method also displays competitive performance in image inversion and editing.
Paper Structure (29 sections, 17 equations, 21 figures, 3 tables, 3 algorithms)

This paper contains 29 sections, 17 equations, 21 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries: Flow-based models and Rectified Flow
  3. Methods
  4. RFDS
  5. iRFDS
  6. RFDS-Rev
  7. Applying iRFDS and RFDS-Rev to Diffusion Models
  8. Classifier Free Guidance (CFG)
  9. Experiments
  10. RFDS vs. RFDS-Rev vs. Diffusion Priors vs. Diffusion RFDS-Rev
  11. iRFDS vs. Diffusion Methods
  12. Ablation Experiments
  13. Summary of observations and Discussion
  14. Conclusion, Limitations
  15. Broader Impacts.
  16. ...and 14 more sections

Figures (21)

  • Figure 1: Demonstration of potential use cases of the proposed rectified flow priors. Our methods can be used on 2D inversion and editing and text-to-3D generation.
  • Figure 2: Illustration of our proposed methods.
  • Figure 3: 2D Comparison. Text Prompts: "A castle next to a mountain", "An astronaut is riding a horse" and "A monster truck".
  • Figure 4: 3D Comparison. Text Prompts: "An antique wooden rocking horse", "A bear holding a sign that says Hello world" and "Hot popcorn jump out from the red striped popcorn maker".
  • Figure 5: Comparison of 2D editing. Our iRFDS method is able to perform prompt-faithful editing.
  • ...and 16 more figures