Stream-R1: Reliability-Perplexity Aware Reward Distillation for Streaming Video Generation

Abstract

Distillation-based acceleration has become foundational for making autoregressive streaming video diffusion models practical, with distribution matching distillation (DMD) as the de facto choice. Existing methods, however, train the student to match the teacher's output indiscriminately, treating every rollout, frame, and pixel as equally reliable supervision. We argue that this caps distilled quality, since it overlooks two complementary axes of variance in DMD supervision: Inter-Reliability across student rollouts whose supervision varies in reliability, and Intra-Perplexity across spatial regions and temporal frames that contribute unequally to where quality can still be improved. The objective thus conflates two questions under a uniform weight: whether to learn from each rollout, and where to concentrate optimization within it. To address this, we propose Stream-R1, a Reliability-Perplexity Aware Reward Distillation framework that adaptively reweights the distillation objective at both rollout and spatiotemporal-element levels through a single shared reward-guided mechanism. At the Inter-Reliability level, Stream-R1 rescales each rollout's loss by an exponential of a pretrained video reward score, so that rollouts with reliable supervision dominate optimization. At the Intra-Perplexity level, it back-propagates the same reward model to extract per-pixel gradient saliency, which is factored into spatial and temporal weights that concentrate optimization pressure on regions and frames where refinement yields the largest expected gain. An adaptive balancing mechanism prevents any single quality axis from dominating across visual quality, motion quality, and text alignment. Stream-R1 attains consistent improvements on all three dimensions over distillation baselines on standard streaming video generation benchmarks, without architectural modification or additional inference cost.

Figures (5)

• Figure 1: Motivation of Stream-R1. (a) The DMD supervision signal exhibits two complementary axes of variance: Inter-Reliability across different rollouts, and Intra-Perplexity across spatiotemporal regions within each rollout. (b) The existing DMD paradigm assigns uniform sample preference to all rollouts and uniform optimization intensity to all regions, regardless of their reliability or perplexity. (c) Our reliability-perplexity aware DMD upweights rollouts on which the supervision is reliable and concentrates higher optimization intensity on regions where further refinement yields the largest expected gain, all driven by a single reward model.
• Figure 2: Overview of Stream-R1. (a) The fake rollout from $G_\theta$ is scored by DMD networks $f_{\text{fake}}, f_{\text{Real}}$ and the Stream R1 module; the distillation signal is modulated by an Inter-Reliability weight $w_{\text{inter}}$ and an intra-instance weight $\mathbf{W}_{\text{intra}}$ to form $\mathcal{L}_{\text{Stream-R1}} = \mathbf{W}_{\text{inter}} \cdot (W_{\text{intra}} \odot \mathcal{L}_{\text{DMD}})$. Bottom: Inside the Stream R1 module, (b) Inter-Reliability Score Extraction produces a scalar reward $R_{\text{score}}$ for $w_{\text{inter}}$ and per-axis saliencies $s_{\text{VQ/MQ/TA}}$; (c) Adaptive Gradient-Saliency Combination fuses the three saliencies into a unified map; (d) Spatiotemporal Decomposition factorizes the map into spatial and temporal weights to form $W_{\text{intra}}$. A single reward model drives both weights.
• Figure 3: Qualitative comparison on long video generation. For each pair, the top row is Reward Forcing and the bottom row is Stream-R1.
• Figure 4: Per-metric quality comparison at varying video lengths. Stream-R1 (blue) consistently outperforms Reward Forcing (orange) across all six metrics at every duration. The advantage widens as video length increases, particularly at 120s and 180s, confirming that spatiotemporal reward-guided weighting mitigates the quality drift accumulated during long autoregressive rollouts.
• Figure 5: Spatiotemporal saliency under controlled degradation. Gaussian blur is injected only into the lower half of each sampled frame so that every frame itself forms a clean (top) versus degraded (bottom) contrast; the blurred area further expands across the four frames from left to right. Top: reward-model gradient saliency overlaid on the degraded frames. Middle: the degraded frames, where only the lower half is corrupted. Bottom: per-frame temporal weights $w_t$, growing as the degraded area enlarges.