Burst Image Super-Resolution with Base Frame Selection

TL;DR

This work explores using burst shots with non-uniform exposures to confront real-world practical scenarios by introducing a new benchmark dataset that includes the burst frames at varying exposure times to obtain a broader range of irradiance and motion characteristics within a scene.

Abstract

Burst image super-resolution has been a topic of active research in recent years due to its ability to obtain a high-resolution image by using complementary information between multiple frames in the burst. In this work, we explore using burst shots with non-uniform exposures to confront real-world practical scenarios by introducing a new benchmark dataset, dubbed Non-uniformly Exposed Burst Image (NEBI), that includes the burst frames at varying exposure times to obtain a broader range of irradiance and motion characteristics within a scene. As burst shots with non-uniform exposures exhibit varying levels of degradation, fusing information of the burst shots into the first frame as a base frame may not result in optimal image quality. To address this limitation, we propose a Frame Selection Network (FSN) for non-uniform scenarios. This network seamlessly integrates into existing super-resolution methods in a plug-and-play manner with low computational costs. The comparative analysis reveals the effectiveness of the nonuniform setting for the practical scenario and our FSN on synthetic-/real- NEBI datasets.

Figures (5)

• Figure 1: Output quality decreases when a BISR model uses burst shots taken at sub-optimal exposure times compared to optimal time (c). On the other hand, as shown in (f), the model reconstructs a high-resolution (HR) image as if it were captured using optimally exposed burst shots by harnessing information from non-uniformly exposed burst shots.
• Figure 2: (a) Previous methods bhat2021deepbhat2021deep_reparadudhane2022burst fix the base frame to the first frame, which often suffers from degradation (e.g., noise, blur), resulting in poor image quality. (b) Our approach dynamically selects the base frame using a frame selection network, enhancing the quality of the final output image. This adaptive strategy helps the model generate a cleaner final output image.
• Figure 3: Visualization of the Synthetic-/Real-NEBI dataset. The left four columns display a subset of 14 input bursts, while the rightmost column presents the ground-truth image corresponding to the first input frame. From left to right in the burst sequence, the exposure time increases, resulting in decreased noise and increased blur. Best view to zoom.
• Figure 4: (a) Overall architecture of frame selector. The network begins with a feature extractor followed by $L$ Contextual Motion Aggregation (CMA) blocks, each containing appearance and motion streams that produce image feature $F_L$ and motion feature $M_L$. These features are summed and further refined through a Global-Averaging-Pooling (GAP) layer and an MLP layer to predict the likelihood of being the base frame. (b) Feature Correlation Module (FCM). Given the previous motion feature $M_{l-1}$, our FCM generates the motion information $C_l$ by computing the local feature correlation. The resultant $F_l$ and $M_l$ are obtained by propagating $C_l$.
• Figure 5: Qualitative comparison on synthetic-NEBI (top two rows) and real-NEBI (bottom row). BIPNet dudhane2022burst with our frame selector enhances high-frequency image details by merging effectively complementary information from multiple frames into the selected base frame, while BIPNet dudhane2022burst predicts blurry images due to the severe degradations in the first frame. Best view to zoom.