Learning to Refocus with Video Diffusion Models

TL;DR

This work tackles the problem of post-capture refocusing from a single defocused image by leveraging video diffusion priors to generate a full focal stack as a coherent video sequence. It reframes refocusing as a multi-frame generation task and introduces position-aware modified classifier-free guidance to produce realistic focal stacks in one pass. A large-scale, real-world focal stack dataset of 1,637 iPhone captures is released to support training and benchmarking, and the method demonstrates strong perceptual quality, robustness, and cross-device generalization, enabling downstream DoF editing and motion deblurring. The approach advances practical, accessible refocusing capabilities for everyday photography and opens avenues for further improvements with larger-aperture data and sensor-informed conditioning.

Abstract

Focus is a cornerstone of photography, yet autofocus systems often fail to capture the intended subject, and users frequently wish to adjust focus after capture. We introduce a novel method for realistic post-capture refocusing using video diffusion models. From a single defocused image, our approach generates a perceptually accurate focal stack, represented as a video sequence, enabling interactive refocusing and unlocking a range of downstream applications. We release a large-scale focal stack dataset acquired under diverse real-world smartphone conditions to support this work and future research. Our method consistently outperforms existing approaches in both perceptual quality and robustness across challenging scenarios, paving the way for more advanced focus-editing capabilities in everyday photography. Code and data are available at https://learn2refocus.github.io

Figures (11)

• Figure 1: Illustration showing that video diffusion models have priors on generating focal stacks. The video sequence is generated by prompting a model firefly to generate "a focus pull of a bug on a leaf".
• Figure 2: Eight sample focal stacks from our dataset consisting of 1637 total scenes. More focal stacks are visualized in a supplementary material montage.
• Figure 3: The original focal stack exhibits misalignment caused by focal breathing, evident from the noticeable edge movements. Our pre-processing corrects these distortions, resulting in a well-aligned stack that enables more accurate refocusing.
• Figure 4: Illustration of our modified classifier-free guidance for a focal stack with $F=3$ frames. During training, the latent corresponding to a randomly selected focal position is passed to the video diffusion model at the matching position, while all other conditioning inputs are set to zero.
• Figure 5: Refocusing results from various input focal positions to output focal positions. We compare our method to RGAN-NAF refocusgan, NAFNet naf, and ground truth (GT). Our method reconstructs coarse structures in the pine-cone (third row) and fine details in hair (second row).