Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reward-Forcing: Autoregressive Video Generation with Reward Feedback

Jingran Zhang, Ning Li, Yuanhao Ban, Andrew Bai, Justin Cui

TL;DR

Reward-Forcing introduces an autoregressive video generation framework guided by a differentiable reward model to improve texture quality while preserving motion, without heavy reliance on a bidirectional teacher. The method initializes motion with an ODE-based trajectory derived from a teacher, then applies reward-driven refinement to enhance appearance, supervising primarily the last frame to maintain dynamics. On VBench, it achieves competitive quality and total scores relative to autoregressive baselines and even surpasses some bidirectional models of similar size, suggesting reward signals can compensate for limited teacher guidance. This data-efficient, scalable approach decouples motion learning from texture refinement and offers customizable video generation via reward models, with potential for broader streaming and real-time applications.

Abstract

While most prior work in video generation relies on bidirectional architectures, recent efforts have sought to adapt these models into autoregressive variants to support near real-time generation. However, such adaptations often depend heavily on teacher models, which can limit performance, particularly in the absence of a strong autoregressive teacher, resulting in output quality that typically lags behind their bidirectional counterparts. In this paper, we explore an alternative approach that uses reward signals to guide the generation process, enabling more efficient and scalable autoregressive generation. By using reward signals to guide the model, our method simplifies training while preserving high visual fidelity and temporal consistency. Through extensive experiments on standard benchmarks, we find that our approach performs comparably to existing autoregressive models and, in some cases, surpasses similarly sized bidirectional models by avoiding constraints imposed by teacher architectures. For example, on VBench, our method achieves a total score of 84.92, closely matching state-of-the-art autoregressive methods that score 84.31 but require significant heterogeneous distillation.

Reward-Forcing: Autoregressive Video Generation with Reward Feedback

TL;DR

Reward-Forcing introduces an autoregressive video generation framework guided by a differentiable reward model to improve texture quality while preserving motion, without heavy reliance on a bidirectional teacher. The method initializes motion with an ODE-based trajectory derived from a teacher, then applies reward-driven refinement to enhance appearance, supervising primarily the last frame to maintain dynamics. On VBench, it achieves competitive quality and total scores relative to autoregressive baselines and even surpasses some bidirectional models of similar size, suggesting reward signals can compensate for limited teacher guidance. This data-efficient, scalable approach decouples motion learning from texture refinement and offers customizable video generation via reward models, with potential for broader streaming and real-time applications.

Abstract

While most prior work in video generation relies on bidirectional architectures, recent efforts have sought to adapt these models into autoregressive variants to support near real-time generation. However, such adaptations often depend heavily on teacher models, which can limit performance, particularly in the absence of a strong autoregressive teacher, resulting in output quality that typically lags behind their bidirectional counterparts. In this paper, we explore an alternative approach that uses reward signals to guide the generation process, enabling more efficient and scalable autoregressive generation. By using reward signals to guide the model, our method simplifies training while preserving high visual fidelity and temporal consistency. Through extensive experiments on standard benchmarks, we find that our approach performs comparably to existing autoregressive models and, in some cases, surpasses similarly sized bidirectional models by avoiding constraints imposed by teacher architectures. For example, on VBench, our method achieves a total score of 84.92, closely matching state-of-the-art autoregressive methods that score 84.31 but require significant heterogeneous distillation.
Paper Structure (33 sections, 7 equations, 4 figures, 3 tables)

This paper contains 33 sections, 7 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Diffusion Models for Image Generation
  4. Video Diffusion Models
  5. Diffusion Acceleration
  6. Autoregressive and Streaming Generation
  7. Reinforcement Learning for Generative Models
  8. Motion in Video Diffusion Models
  9. Methodology
  10. Preliminaries
  11. Diffusion Model
  12. Video Diffusion Distillation
  13. Training with ODE and Reward Guidance
  14. Optimization with Reward Feedback
  15. Experimental Results
  16. ...and 18 more sections

Figures (4)

  • Figure 1: Overview of our proposed pipeline. The method first leverages a small set of ODE-based trajectories generated by a teacher model to guide the learning of motion dynamics. Subsequently, a reward model is employed to enhance the generation with fine-grained texture details.
  • Figure 2: Comparison between our methods and baseline methods on selected VBench metrics. Our method shows competitive performances without extensive heterogeneous distillation.
  • Figure 3: Comparison of videos generated by our method and other baseline methods, using the prompt: "A joyful, playful Corgi running and frolicking in a vibrant park during sunset." The second row displays the optical flow of the video generated by the model trained solely with ODE trajectories, highlighting the motion patterns the model has learned.
  • Figure 4: More generated samples. Prompts are randomly sampled from VBench of various scenes to benchmark the overall capability of the model.