Pretraining Frame Preservation in Autoregressive Video Memory Compression

TL;DR

The paper tackles the problem of sustaining long-range temporal coherence in autoregressive video generation by reducing the history context to a compact, detail-rich memory. It introduces PFP, a pretraining scheme for a memory compression model that is optimized to retrieve high-frequency frame details at arbitrary times within a long video history, compressing 20+ seconds into about 5k latent context. The approach first pretrains the compressor with a frame-retrieval objective across randomized frame indices, then finetunes it as a history memory encoder for diffusion-based autoregressive video generation (e.g., DiT Wan/HunyuanVideo) using LoRA adapters. Extensive ablations, qualitative analyses, and quantitative metrics (PSNR, SSIM, LPIPS, VBench) demonstrate that the pretrained encoder improves temporal consistency and frame fidelity, enabling long-horizon video generation with substantially reduced context cost. The work offers pretrained components and design insights for practical long-form video synthesis and editing, balancing context length and quality through explicit retrieval-focused pretraining.

Abstract

We present PFP, a neural network structure to compress long videos into short contexts, with an explicit pretraining objective to preserve the high-frequency details of single frames at arbitrary temporal positions. The baseline model can compress a 20-second video into a context at about 5k length, where random frames can be retrieved with perceptually preserved appearances. Such pretrained models can be directly fine-tuned as memory encoders for autoregressive video models, enabling long history memory with low context cost and relatively low fidelity loss. We evaluate the framework with ablative settings and discuss the trade-offs of possible neural architecture designs.

Figures (10)

• Figure 1: This video was generated autoregressively second-by-second with full history context (without cutting out any history frames). The 20+ second history was compressed into a $\sim$5k context length and processed by an RTX 4070 12GB.
• Figure 2: Pretraining of memory compression models. The memory compression model has to compress long videos (e.g., 20 seconds) into short contexts (e.g., of length 5k). The objective of the pretraining is to retrieve frames with high-frequency details in arbitrary history time positions.
• Figure 3: Architecture of memory compression model. We use 3D convolution, SiLU, and attention to establish a lightweight neural structure as the baseline compression model. Different alternative architectures (e.g., various channels, full transformer, etc.) are possible and will be discussed in ablation.
• Figure 4: Finetuning autoregressive video models. We illustrate the finetuning and inference of the final autoregressive video models. The pretraining of the memory compression model is finished before the finetuning.
• Figure 5: Visual comparison on compression reconstruction. We present reconstruction results after the pretraining using different possible neural structures and various compression settings. * The "Large Patchifier" is technically equivalent to zhang2025framepack.