CanvasMAR: Improving Masked Autoregressive Video Generation With Canvas

TL;DR

CanvasMAR tackles slow-start and error accumulation in masked autoregressive video generation by introducing a canvas that provides a blurred, global forecast of the next frame. A temporal ViT and a fast Canvas ViT generate this canvas to condition a spatial MAR, enabling coherent frame synthesis with fewer autoregressive steps. Compositional classifier-free guidance and noise-based canvas augmentation further improve robustness to artifacts and temporal-spatial consistency. Evaluations on BAIR and Kinetics-600 show CanvasMAR achieving competitive or superior performance among autoregressive methods and rivaling diffusion-based approaches, while maintaining interactive generation with reduced steps.

Abstract

Masked autoregressive models (MAR) have recently emerged as a powerful paradigm for image and video generation, combining the flexibility of masked modeling with the potential of continuous tokenizer. However, video MAR models suffer from two major limitations: the slow-start problem, caused by the lack of a structured global prior at early sampling stages, and error accumulation across the autoregression in both spatial and temporal dimensions. In this work, we propose CanvasMAR, a novel video MAR model that mitigates these issues by introducing a canvas mechanism--a blurred, global prediction of the next frame, used as the starting point for masked generation. The canvas provides global structure early in sampling, enabling faster and more coherent frame synthesis. Furthermore, we introduce compositional classifier-free guidance that jointly enlarges spatial (canvas) and temporal conditioning, and employ noise-based canvas augmentation to enhance robustness. Experiments on the BAIR and Kinetics-600 benchmarks demonstrate that CanvasMAR produces high-quality videos with fewer autoregressive steps. Our approach achieves remarkable performance among autoregressive models on Kinetics-600 dataset and rivals diffusion-based methods.

Figures (7)

• Figure 1: Video generation with Masked Autoregressive Models using only 2 autoregressive steps per $16 \times 16$ frame. Frames with red borders (left) denote the conditioning inputs. Without a canvas (top row), the model quickly loses global coherence. Our approach (second and bottom rows) mitigates this by first predicting a blurred canvas (shown in the second row with blue borders) that captures coarse motion and global structure. The canvas then replaces the uniform mask in the masked autoregressive model, allowing global fidelity to be preserved even under an aggressive sampling regime.
• Figure 2: Left: Inference procedure of CanvasMAR. The framework consists of three modules: Temporal ViT, Canvas ViT, and Spatial MAR. The Temporal ViT encodes the historical frames and produces a temporal embedding that serves as a temporal condition for both the Canvas ViT and the Spatial MAR. The Canvas ViT takes the temporal embedding together with the most recent frame to predict a coarse canvas, i.e., the initial guess of future frame, which along with the temporal embedding provides spatial conditioning for the Spatial MAR. Right: Training procedure of the Spatial MAR. Using a Spatial ViT, it learns to reconstruct randomly masked regions within frames. Unlike vanilla MAR (a), which employs a uniform learnable mask embedding for all areas, CanvasMAR replaces this with the predicted canvas (b), offering a more informative and adaptive conditioning signal.
• Figure 3: Ablation studies on BAIR. (\ref{['fig:effect_canvas']}) Effectiveness of the proposed canvas mechanism. (\ref{['fig:effect_cfg']}) Effectiveness of the compositional classifier-free guidance.
• Figure 4: Qualitative results demonstrating the effectiveness of the canvas on BAIR ($16\times16$ latents). The frames with the red border indicates the conditioning frames, while the remaining frames are model predictions with only two autoregressive steps. a) Results from the model with the canvas; b) results without it.
• Figure 5: Left: Qualitative results demonstrating the effectiveness of the guidance on Kinetics-600. Frames with red borders indicate the conditioning frames. Without any guidance, the video tends to exhibit distortions in complex kinetics. Temporal guidance improves frame quality and temporal consistency, albeit at the cost of motion intensity. Spatial (Canvas) guidance preserves motion but may occasionally amplify flaws in the predicted canvas (e.g., the legs). When both types of guidance are combined, motion and frame quality reach their optimal levels. Right: Qualitative results of the generation process. a) The predicted canvas. b) The generated frame conditioned on the canvas.