BurstM: Deep Burst Multi-scale SR using Fourier Space with Optical Flow

TL;DR

BurstM tackles misalignment and limited high-frequency recovery in multi-frame SR by combining optical-flow-based frame alignment with Fourier-space warping and continuous Fourier coefficient prediction. The approach uses an Learnable Neural Warping module to generate Fourier-aware features, a Blender+MLP reconstruction path, and an INR-inspired multi-scale SR capability that supports scales $\ imes2, \ imes3, \ imes4$ in a single unimodel. Experimental results on SyntheticBurst and BurstSR show state-of-the-art PSNR/SSIM, reduced boundary artifacts, and faster inference compared with DCN-based methods, validating both accuracy and efficiency. The work highlights the practical impact of integrating optical flow, Fourier feature representations, and implicit neural representations for flexible, high-quality MFSR.

Abstract

Multi frame super-resolution(MFSR) achieves higher performance than single image super-resolution (SISR), because MFSR leverages abundant information from multiple frames. Recent MFSR approaches adapt the deformable convolution network (DCN) to align the frames. However, the existing MFSR suffers from misalignments between the reference and source frames due to the limitations of DCN, such as small receptive fields and the predefined number of kernels. From these problems, existing MFSR approaches struggle to represent high-frequency information. To this end, we propose Deep Burst Multi-scale SR using Fourier Space with Optical Flow (BurstM). The proposed method estimates the optical flow offset for accurate alignment and predicts the continuous Fourier coefficient of each frame for representing high-frequency textures. In addition, we have enhanced the network flexibility by supporting various super-resolution (SR) scale factors with the unimodel. We demonstrate that our method has the highest performance and flexibility than the existing MFSR methods. Our source code is available at https://github.com/Egkang-Luis/burstm

Figures (10)

• Figure 1: Overview of proposed Burst SR network. The proposed method (BurstM) predicts high-resolution images from multiple low-resolution images. The neural warping performs high-precision warping with upscaling on Fourier space. Additionally, the neural warping contributes to the prediction of an accurate high-resolution images.
• Figure 2: Proposed BurstM Network. The network estimates optical flow between the reference and source frames using FNet. The neural warping $\mathbf{T_{\boldsymbol{\nu}}}$ emphasizes the high-frequency details and implements the warping on Fourier space. Reconstruction module $\mathbf{R_{\boldsymbol{\varphi}}}$ includes the blender $\mathbf{B_{\boldsymbol{\eta}}}$ and the decoder $\mathbf{G_{\boldsymbol{\Theta}}}$. The blender $\mathbf{B_{\boldsymbol{\eta}}}$ matches color differences and merges multiple frames into a single frame. The Decoder $\mathbf{G_{\boldsymbol{\Theta}}}$ predicts the final output. The gray color indicates a single operation such a convolution and LeakyReLU activation and the others include nonlinear activation
• Figure 3: Visual comparison for $\times$4 super-resolution for BurstSR. BurstM shows clearer textures and correct color than the others without the boundary artifacts.
• Figure 4: Visual comparison of multi-scale SR on BurstSR. BurstM predict several scaled-images with unimodel and the others are downsampling after $\times$4 SR.
• Figure 5: Visual comparison for $\times$4 Super-Resolution for SynteticBurst.