LVTINO: LAtent Video consisTency INverse sOlver for High Definition Video Restoration

TL;DR

This work introduces LATINO, a zero-shot, plug-and-play inverse solver for high-definition video restoration that leverages priors encoded by Video Consistency Models (VCMs) and Latent Consistency Models (LCMs). It formulates a product-of-experts prior combining a VCM, an ICM, and a spatio-temporal TV regularizer, and performs gradient-free posterior sampling via a split Langevin scheme with stochastic autoencoder steps and Moreau–Yosida proximal updates. LATINO achieves strong measurement-consistency and perceptual quality with only a handful of neural function evaluations, outperforms frame-by-frame LDM-based baselines like VISION-XL, and offers favorable memory characteristics by avoiding backpropagation through large video diffusion models. The experimental results on the Adobe240-based HD video restoration tasks demonstrate substantial improvements in FVMD, LPIPS, and PSNR/SSIM across challenging degradations, establishing a new benchmark for joint spatial-temporal video restoration that is both accurate and computationally efficient.

Abstract

Computational imaging methods increasingly rely on powerful generative diffusion models to tackle challenging image restoration tasks. In particular, state-of-the-art zero-shot image inverse solvers leverage distilled text-to-image latent diffusion models (LDMs) to achieve unprecedented accuracy and perceptual quality with high computational efficiency. However, extending these advances to high-definition video restoration remains a significant challenge, due to the need to recover fine spatial detail while capturing subtle temporal dependencies. Consequently, methods that naively apply image-based LDM priors on a frame-by-frame basis often result in temporally inconsistent reconstructions. We address this challenge by leveraging recent advances in Video Consistency Models (VCMs), which distill video latent diffusion models into fast generators that explicitly capture temporal causality. Building on this foundation, we propose LVTINO, the first zero-shot or plug-and-play inverse solver for high definition video restoration with priors encoded by VCMs. Our conditioning mechanism bypasses the need for automatic differentiation and achieves state-of-the-art video reconstruction quality with only a few neural function evaluations, while ensuring strong measurement consistency and smooth temporal transitions across frames. Extensive experiments on a diverse set of video inverse problems show significant perceptual improvements over current state-of-the-art methods that apply image LDMs frame by frame, establishing a new benchmark in both reconstruction fidelity and computational efficiency.

Figures (9)

• Figure 1: Results on joint spatial-temporal super-resolution by factor $\times 8$.
• Figure 2: One step of the Lorigin=c]180A TINO solver, a discretization of the Langevin SDE (\ref{['eq:split-Langevin-video-4']}) which targets the posterior $p({\bm{x}}|{\bm{y}},c,\lambda)$, involving two stochastic autoencoding (SAE) steps and two proximal steps.
• Figure 3: Comparison between slices from 81 consecutive frames for Problem C (seq. C2). Slice images $(i,\tau)$ are obtained from the video tensor $(i,j,\tau)$ by fixing a column index $j$ shown in green.
• Figure 4: Visual comparison for Problem A (seq. A1). The continuity of the motion is retrieved as the hand moves from right to left. See full videos: https://streamable.com/hpoijy and https://streamable.com/mptmed.
• Figure 5: Visual comparison for Problem B (seq. B2). The flickering problem is solved by Lorigin=c]180A TINO (see darker and lighter area behind the chair). See full videos: https://streamable.com/qphrea and https://streamable.com/o6p8px.