Generating the Past, Present and Future from a Motion-Blurred Image

TL;DR

This paper reframes motion blur as a cue for scene dynamics by conditioning a large-scale video diffusion model on a motion-blurred image and explicit per-frame exposure intervals to synthesize past, present, and future video frames. By fine-tuning a pre-trained video diffusion transformer with latent-space encodings and specialized exposure-time conditioning, the method achieves state-of-the-art performance in present-frame generation and robustly extends to past and future frames, enabling downstream tasks like 3D reconstruction and camera-trajectory recovery. The approach generalizes to challenging wild scenes and even historical photographs, offering new capabilities for 4D scene understanding and dynamic human pose estimation from a single blurred image. The work also explores exposure-interval control, multi-modality, and 3D-consistency diagnostics, highlighting both the potential of large video priors for inverse imaging and the limitations when blur deviates from the assumed model.

Abstract

We seek to answer the question: what can a motion-blurred image reveal about a scene's past, present, and future? Although motion blur obscures image details and degrades visual quality, it also encodes information about scene and camera motion during an exposure. Previous techniques leverage this information to estimate a sharp image from an input blurry one, or to predict a sequence of video frames showing what might have occurred at the moment of image capture. However, they rely on handcrafted priors or network architectures to resolve ambiguities in this inverse problem, and do not incorporate image and video priors on large-scale datasets. As such, existing methods struggle to reproduce complex scene dynamics and do not attempt to recover what occurred before or after an image was taken. Here, we introduce a new technique that repurposes a pre-trained video diffusion model trained on internet-scale datasets to recover videos revealing complex scene dynamics during the moment of capture and what might have occurred immediately into the past or future. Our approach is robust and versatile; it outperforms previous methods for this task, generalizes to challenging in-the-wild images, and supports downstream tasks such as recovering camera trajectories, object motion, and dynamic 3D scene structure. Code and data are available at https://blur2vid.github.io

Figures (11)

• Figure 1: When given a motion-blurred image of a soccer ball as input, an off-the-shelf video diffusion model runway2025gen4 generates a video that is consistent with the direction of motion blur. In this example, the model correctly predicts the soccer ball's horizontal or vertical motion when given the same text prompt "a soccer ball moving without spinning."
• Figure 2: Model overview. Our approach takes as input a motion-blurred image $I\xspace_{\mathcal{T}\xspace}$. We encode it into the latent space of a pre-trained video diffusion transformer and concatenate the resulting latent image $\tilde{I}\xspace_{\mathcal{T}\xspace}$ with a set of noisy latent video frames $\tilde{V}\xspace$. The latent video frames are associated with exposure times $\tilde{\mathcal{T}}\xspace_{1}, \ldots, \tilde{\mathcal{T}}\xspace_{\tilde{F}\xspace}$, which control the exposures of the video frames we seek to generate, i.e., in the past, present, or future relative to the motion-blurred image. Each latent video frame encodes multiple output video frames and so is associated with multiple exposure times. We modulate the latent frames using position encoding su2024roformer to incorporate information about each latent pixel's spatial position and frame index. The exposure times are encoded using a sinusoidal encoding and linear projection, and we add the result to the latent frames. The resulting latent frames are patchified, passed as input to the video model, and the model generates denoised latent frames that are decoded to recover the output video. Photos from the GoPro dataset nah2017deep
• Figure 3: PSNR per frame (using best ordering of frames) when predicting 13 frames from a blurry image synthesized from 7 frames in the GoPro dataset.
• Figure 4: (a--b) Results on the GoPro and B-AIST++ datasets nah2017deepzhong2022animation. We compare our method to MotionETR zhang2021exposure, Jin et al. jin2018learning, and Animation from Blur zhong2022animation and find that our method recovers significantly clearer output video frames with motion tracks that are more consistent with the ground-truth video sequence (tracks estimated using Cotracker karaev2024cotracker). (c--d) Additional baseline comparisons to in-the-wild data. We find that Jin et al. jin2018learning and MotionETR fail to recover sharp video frames on these challenging, in-the-wild sequences, likely due to the more limited scale of their training datasets and learned motion priors (compare to our results on these scenes in Figures \ref{['fig:teaser']} and \ref{['fig:applications']}).
• Figure 5: Applications of the proposed method (please see the video results in the supplemental webpage). (a) We demonstrate generating the past, present, and future from in-the-wild motion-blurred images, as indicated by the red, green, and blue labels. The method recovers sharp frames and scene dynamics from a mixer (top) and a busy city street (bottom). Note the complex motion trajectories recovered by applying an off-the-shelf tracker karaev2024cotracker to our generated videos. (b) By exploiting motion blur in historical photos, we reveal scene dynamics, e.g., the movement of Mohammad Ali in a boxing match or astronaut John Glenn picking up a camera. (c) We bring images "to life" by predicting 3D scene dynamics and camera poses from our generated video frames with off-the-shelf structure from motion methods Li2024megasam. (d) We even recover subtle motions in black and white photographs captured during World War II over 80 years ago. We reveal motions in the generated deblurred frames (insets) by applying an optical flow method teed2020raft to a past and future predicted frame. (e) Finally, we recover 3D facial dynamics from a motion-blurred image by applying a face tracker to our output video taubner20243d. Historical photos: (1) Coast Guard Lands the British Marines (1944), U.S. National Archives and Records, public domain; (2) Senator John Glenn (1998), NASA, public domain; (3) Mohammad Ali boxing J√ºrgen Blin (1971), © Wikimedia Commons, CC BY-SA 4.0. (4) Blurry face, © DieselDemon, Flickr, CC BY 2.0.