StableMotion: Repurposing Diffusion-Based Image Priors for Motion Estimation

TL;DR

StableMotion demonstrates that pretrained diffusion priors can be repurposed to perform motion estimation for single-image rectification tasks, specifically Stitched Image Rectangling and Rolling Shutter Correction. By adapting a VAE for flow refinement and a UNet-based motion estimator, and introducing Sampling Steps Disaster (SSD) and Adaptive Ensemble Strategy (AES), the method achieves one-step inference with high fidelity. The approach delivers state-of-the-art results on public benchmarks and enables up to 200x speedups over prior diffusion-based methods due to SSD and efficient design. The work also highlights strong generalization to unseen distributions, underscoring the practical benefits of leveraging diffusion priors for geometric vision tasks while reducing training costs.

Abstract

We present StableMotion, a novel framework leverages knowledge (geometry and content priors) from pretrained large-scale image diffusion models to perform motion estimation, solving single-image-based image rectification tasks such as Stitched Image Rectangling (SIR) and Rolling Shutter Correction (RSC). Specifically, StableMotion framework takes text-to-image Stable Diffusion (SD) models as backbone and repurposes it into an image-to-motion estimator. To mitigate inconsistent output produced by diffusion models, we propose Adaptive Ensemble Strategy (AES) that consolidates multiple outputs into a cohesive, high-fidelity result. Additionally, we present the concept of Sampling Steps Disaster (SSD), the counterintuitive scenario where increasing the number of sampling steps can lead to poorer outcomes, which enables our framework to achieve one-step inference. StableMotion is verified on two image rectification tasks and delivers state-of-the-art performance in both, as well as showing strong generalizability. Supported by SSD, StableMotion offers a speedup of 200 times compared to previous diffusion model-based methods.

Figures (31)

• Figure 1: Two applications of the StableMotion framework, as well as the Sampling Steps Disaster (SSD).
• Figure 2: Repurposing from SD. Taking image rectangling as example. At each timestep, the predicted flow feature is decoded and denormalized into the pixel-space to perform warping and to construct conditional losses.
• Figure 3: Overview of the inference scheme. Taking image rectangling as example. Note that the sampling process is one-step.
• Figure 4: Explanation of Sampling Steps Disaster (SSD). With more than one target distributions, directly performing multiple steps inference yields error.
• Figure 5: Comparing image rectangling (above the dotted line) and rolling shutter correction (below the dotted line) with previous methods. Green arrows points to artifacts or unsolved margins, and the red dashed boxes correspond to the area in the heatmap. The brighter areas on the heatmap indicate a greater discrepancy between the output and the ground truth.