HDRFlow: Real-Time HDR Video Reconstruction with Large Motions

TL;DR

HDRFlow is the first real-time HDR video reconstruction method for video sequences captured with alternating exposures, capable of processing 720p resolution inputs at 25ms, and outperforms previous methods on standard benchmarks.

Abstract

Reconstructing High Dynamic Range (HDR) video from image sequences captured with alternating exposures is challenging, especially in the presence of large camera or object motion. Existing methods typically align low dynamic range sequences using optical flow or attention mechanism for deghosting. However, they often struggle to handle large complex motions and are computationally expensive. To address these challenges, we propose a robust and efficient flow estimator tailored for real-time HDR video reconstruction, named HDRFlow. HDRFlow has three novel designs: an HDR-domain alignment loss (HALoss), an efficient flow network with a multi-size large kernel (MLK), and a new HDR flow training scheme. The HALoss supervises our flow network to learn an HDR-oriented flow for accurate alignment in saturated and dark regions. The MLK can effectively model large motions at a negligible cost. In addition, we incorporate synthetic data, Sintel, into our training dataset, utilizing both its provided forward flow and backward flow generated by us to supervise our flow network, enhancing our performance in large motion regions. Extensive experiments demonstrate that our HDRFlow outperforms previous methods on standard benchmarks. To the best of our knowledge, HDRFlow is the first real-time HDR video reconstruction method for video sequences captured with alternating exposures, capable of processing 720p resolution inputs at 25ms.

Figures (12)

• Figure 1: Row 2: the optical flow from frame 0 to frame -1. Row 3: the resulting HDR images. The methods of Kalantari et al. kalantari19 and Chen et al. chen2021hdr struggle to predict accurate optical flow due to large motions, resulting in ghosting artifacts in the HDR output. In contrast, our HDRFlow predicts HDR-oriented optical flow and exhibits robustness to large motions. We compare our HDR-oriented flow with RAFT's raft flow. RAFT's flow is sub-optimal for HDR fusion, and alignment may fail in occluded regions. leading to significant ghosting artifacts in the HDR output.
• Figure 2: HDR video reconstruction from sequences (image size: $1280\!\times\!720$) captured with three alternating exposures. Row 1 displays four input LDR frames. Rows 2-4 are the reconstructed HDR frames using methods Chen et al. chen2021hdr, LAN-HDR lan_hdr and ours.
• Figure 3: Network architecture of the proposed HDRFlow . We first estimate bidirectional optical flows through the proposed flow network. Then, we align the neighboring frames to the reference frame $t$ based on these estimated flows. To achieve accurate alignment, we introduce a novel HDR-domain alignment loss to supervise our flow network. Finally, the aligned frames and the original frames are fused together through the fusion network to reconstruct a high-quality and ghost-free HDR image for the reference frame.
• Figure 4: Qualitative comparisons with flow-based methods. Left: 3-Exposure scene from the Kalantari13 dataset kalantari13. Right: 2-Exposure scene from the DeepHDRVideo dataset chen2021hdr. Flow visualization is based on the color wheel shown on the corner of the first flow map.
• Figure 5: Effectiveness of HALoss.