STCDiT: Spatio-Temporally Consistent Diffusion Transformer for High-Quality Video Super-Resolution

TL;DR

STCDiT addresses video super-resolution under complex camera motion by coupling a motion-aware VAE reconstruction that segments videos into motion-consistent clips with an anchor-frame guidance mechanism that leverages first-frame latent structure. The approach uses a diffusion transformer with a lightweight, discriminative anchor-frame modulation to propagate structural information while remaining parameter-efficient via LoRA tuning. Quantitative and qualitative evaluations across synthetic and real benchmarks show state-of-the-art performance in both perceptual fidelity (LPIPS, MUSIQ, MANIQA) and temporal stability, with notable gains in PSNR and detailed structural restoration. This method offers a practical, scalable path to high-quality VSR in realistic scenarios with diverse motion, enabling robust video restoration for downstream applications.

Abstract

We present STCDiT, a video super-resolution framework built upon a pre-trained video diffusion model, aiming to restore structurally faithful and temporally stable videos from degraded inputs, even under complex camera motions. The main challenges lie in maintaining temporal stability during reconstruction and preserving structural fidelity during generation. To address these challenges, we first develop a motion-aware VAE reconstruction method that performs segment-wise reconstruction, with each segment clip exhibiting uniform motion characteristic, thereby effectively handling videos with complex camera motions. Moreover, we observe that the first-frame latent extracted by the VAE encoder in each clip, termed the anchor-frame latent, remains unaffected by temporal compression and retains richer spatial structural information than subsequent frame latents. We further develop an anchor-frame guidance approach that leverages structural information from anchor frames to constrain the generation process and improve structural fidelity of video features. Coupling these two designs enables the video diffusion model to achieve high-quality video super-resolution. Extensive experiments show that STCDiT outperforms state-of-the-art methods in terms of structural fidelity and temporal consistency.

Figures (5)

• Figure 1: Comparison of VAE-based reconstruction (a) and diffusion-based restoration (b). Motion-aware reconstruction in (a) allows VAE to handle videos with complex camera motions. Anchor-frame guided enhancement in (b) enables the parameter-efficient fine-tuning of diffusion transformers for better restoration.
• Figure 2: An overview of STCDiT. For VAE reconstruction, we develop a motion-aware strategy (b) that identifies diverse motion patterns in a video and segments it into clips characterized by consistent motion attributes, which are then independently encoded and decoded by the VAE. For anchor-frame guided enhancement, we sparsely select the first-frame latent from $\{\mathbf{X}_{i}\}^{L}_{i=1}$ as anchor-frame latent and employ a feature refinement module (d) to derive their features. Then, the anchor-frame tokens are concatenated with video tokens, which are then fed into the self-attention layer for feature interaction (c). Moreover, an anchor-frame feature modulation module (e) explores useful information from anchor-frame features, facilitating the restoration of anchor-corresponding frame features in video.
• Figure 3: VSR results ($\times 4$) on the real-world benchmark. We provide two types of video motion scenarios: camera shaking and camera zooming. The prompt corresponding to the video of the first row does not contain the word 'condition'. Compared to competing methods, our method restores temporally consistent videos with fidelity structure and vivid details, even under complex inter-frame motions.
• Figure 4: Effectiveness of the motion-aware VAE reconstruction method on a video with camera shaking.
• Figure 5: Effectiveness of the anchor-frame guided enhancement.