Rolling Shutter Correction with Intermediate Distortion Flow Estimation

TL;DR

The paper tackles rolling shutter distortions by directly estimating a distortion flow $U_{r\leftarrow g}$ from GS to RS to recover distortion-free GS frames via backward warping. It introduces a global correlation-based flow attention mechanism to jointly predict GS features and the initial distortion flow, followed by a coarse-to-fine refinement and a multi-distortion flow decoding scheme to handle non-linear and occluded motion. Across synthetic and real RS datasets, the proposed approach achieves state-of-the-art PSNR/SSIM/LPIPS while maintaining high efficiency, outperforming prior methods in both accuracy and speed. This distortion-flow-centric, end-to-end framework offers a robust and scalable paradigm for RS correction applicable to challenging real-world scenes.

Abstract

This paper proposes to correct the rolling shutter (RS) distorted images by estimating the distortion flow from the global shutter (GS) to RS directly. Existing methods usually perform correction using the undistortion flow from the RS to GS. They initially predict the flow from consecutive RS frames, subsequently rescaling it as the displacement fields from the RS frame to the underlying GS image using time-dependent scaling factors. Following this, RS-aware forward warping is employed to convert the RS image into its GS counterpart. Nevertheless, this strategy is prone to two shortcomings. First, the undistortion flow estimation is rendered inaccurate by merely linear scaling the flow, due to the complex non-linear motion nature. Second, RS-aware forward warping often results in unavoidable artifacts. To address these limitations, we introduce a new framework that directly estimates the distortion flow and rectifies the RS image with the backward warping operation. More specifically, we first propose a global correlation-based flow attention mechanism to estimate the initial distortion flow and GS feature jointly, which are then refined by the following coarse-to-fine decoder layers. Additionally, a multi-distortion flow prediction strategy is integrated to mitigate the issue of inaccurate flow estimation further. Experimental results validate the effectiveness of the proposed method, which outperforms state-of-the-art approaches on various benchmarks while maintaining high efficiency. The project is available at \url{https://github.com/ljzycmd/DFRSC}.

Figures (5)

• Figure 1: Model comparison in terms of PSNR (dB), runtime, and model size. The PSNR and runtime are calculated on the Fastec-RS liu2020deep dataset with a resolution of $640 \times 480$ using an RTX 3090 GPU. The proposed method outperforms the state-of-the-art rolling shutter correction methods with higher efficiency.
• Figure 2: (a) Schematic of the undistortion flow and distortion flow. (b) Comparison between the two-stage flow estimation and the proposed direct distortion flow estimation.
• Figure 3: Overview of the proposed method (a) and the detailed architecture of the key components (b), (c). Our model directly predicts the distortion flow for efficient and high-quality RSC.
• Figure 4: Qualitative results comparison against state-of-the-art methods on the synthetic Fastec-RS dataset liu2020deep. Our method removes the RS distortions well and preserves more details in the recovered GS image on such an occluded scene.
• Figure 5: Qualitative results comparison against state-of-the-art methods on the real-world BS-RSC dataset cao2022learning. Our method is effective and robust in recovering the latent GS image accurately from the RS frames distorted by complex non-linear and large motions.