Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inference

Denis Korzhenkov, Adil Karjauv, Animesh Karnewar, Mohsen Ghafoorian, Amirhossein Habibian

TL;DR

This work demonstrates that pretrained video diffusion models can be efficiently transformed into pyramidal pipelines by targeted, low-cost finetuning, significantly reducing inference compute while preserving video quality. It introduces flow matching and distillation losses to guide pyramidal finetuning and extensively studies step-distillation strategies, including distribution matching and adversarial distillation, for both original and pyramidal teachers. The authors also extend the stage-transition theory to broader upsampling schemes and validate a Patch-pyramidal Training variant, showing practical few-step generation with a single high-resolution step. Collectively, these contributions offer a scalable path to efficient video diffusion, enabling near-baseline perceptual quality with substantially reduced FLOPs and latency, and provide insights for future improvements in distillation and patch-based approaches.

Abstract

Recently proposed pyramidal models decompose the conventional forward and backward diffusion processes into multiple stages operating at varying resolutions. These models handle inputs with higher noise levels at lower resolutions, while less noisy inputs are processed at higher resolutions. This hierarchical approach significantly reduces the computational cost of inference in multi-step denoising models. However, existing open-source pyramidal video models have been trained from scratch and tend to underperform compared to state-of-the-art systems in terms of visual plausibility. In this work, we present a pipeline that converts a pretrained diffusion model into a pyramidal one through low-cost finetuning, achieving this transformation without degradation in quality of output videos. Furthermore, we investigate and compare various strategies for step distillation within pyramidal models, aiming to further enhance the inference efficiency. Our results are available at https://qualcomm-ai-research.github.io/PyramidalWan.

PyramidalWan: On Making Pretrained Video Model Pyramidal for Efficient Inference

TL;DR

This work demonstrates that pretrained video diffusion models can be efficiently transformed into pyramidal pipelines by targeted, low-cost finetuning, significantly reducing inference compute while preserving video quality. It introduces flow matching and distillation losses to guide pyramidal finetuning and extensively studies step-distillation strategies, including distribution matching and adversarial distillation, for both original and pyramidal teachers. The authors also extend the stage-transition theory to broader upsampling schemes and validate a Patch-pyramidal Training variant, showing practical few-step generation with a single high-resolution step. Collectively, these contributions offer a scalable path to efficient video diffusion, enabling near-baseline perceptual quality with substantially reduced FLOPs and latency, and provide insights for future improvements in distillation and patch-based approaches.

Abstract

Recently proposed pyramidal models decompose the conventional forward and backward diffusion processes into multiple stages operating at varying resolutions. These models handle inputs with higher noise levels at lower resolutions, while less noisy inputs are processed at higher resolutions. This hierarchical approach significantly reduces the computational cost of inference in multi-step denoising models. However, existing open-source pyramidal video models have been trained from scratch and tend to underperform compared to state-of-the-art systems in terms of visual plausibility. In this work, we present a pipeline that converts a pretrained diffusion model into a pyramidal one through low-cost finetuning, achieving this transformation without degradation in quality of output videos. Furthermore, we investigate and compare various strategies for step distillation within pyramidal models, aiming to further enhance the inference efficiency. Our results are available at https://qualcomm-ai-research.github.io/PyramidalWan.
Paper Structure (29 sections, 34 equations, 2 figures, 3 tables)

This paper contains 29 sections, 34 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Pyramidal models
  4. Patch-pyramidal models
  5. Step distillation of diffusion models
  6. Preliminaries
  7. Stage-wise definition of clean signal
  8. Stage-wise forward process
  9. Method
  10. Pyramidal finetuning
  11. Flow matching loss
  12. Distillation loss
  13. Pyramidal step distillation
  14. DMD with original teacher
  15. DMD with pyramidal teacher
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Inference of different types of models. Left: input and output tensors of a standard DiT always have the same size, and the number of tokens in transformer blocks does not depend on the noise level. Center: in pyramidal flow matching, higher noise levels are processed at smaller spatiotemporal resolution. For transition between stages special corrective noise should be added after upsampling. Right: in PPF framework instead of changing the resolution, kernel size of patchifier is adjusted for each stage. This keeps the number of tokens equal to that in pyramidal flow matching.
  • Figure 2: Examples of video generations. Videos produced by our pyramidal step-distilled model are similar in quality to outputs of more computationally expensive baselines.