Neural Augmentation Based Panoramic High Dynamic Range Stitching

TL;DR

The paper tackles panoramic HDR stitching from geometrically synchronized LDR images captured at different exposures and overlapping fields of view. It introduces a neural augmentation framework that blends a physics-driven IMF-based initialization using Weighted Histogram Averaging (WHA) with a data-driven Multi-Exposure Awareness Network (MEAN) to refine results, followed by multi-scale exposure fusion in the log domain to produce an information-enriched panoramic LDR image. Key contributions include WHA for robust IMF estimation without CRFs, the MEAN architecture with EAGB and SRRGs for high-frequency preservation, and a MEF-based fusion that yields improved artifact suppression and detail recovery. The framework demonstrates faster convergence and superior performance compared to state-of-the-art panoramic stitching methods, offering a practical approach for real-world HDR panoramic imaging from OFOV data.

Abstract

Due to saturated regions of inputting low dynamic range (LDR) images and large intensity changes among the LDR images caused by different exposures, it is challenging to produce an information enriched panoramic LDR image without visual artifacts for a high dynamic range (HDR) scene through stitching multiple geometrically synchronized LDR images with different exposures and pairwise overlapping fields of views (OFOVs). Fortunately, the stitching of such images is innately a perfect scenario for the fusion of a physics-driven approach and a data-driven approach due to their OFOVs. Based on this new insight, a novel neural augmentation based panoramic HDR stitching algorithm is proposed in this paper. The physics-driven approach is built up using the OFOVs. Different exposed images of each view are initially generated by using the physics-driven approach, are then refined by a data-driven approach, and are finally used to produce panoramic LDR images with different exposures. All the panoramic LDR images with different exposures are combined together via a multi-scale exposure fusion algorithm to produce the final panoramic LDR image. Experimental results demonstrate the proposed algorithm outperforms existing panoramic stitching algorithms.

Figures (10)

• Figure 1: Illustration of panoramic LDR and HDR imaging. (a) are three differently exposed LDR images with OFOVs and exposure value (EV) gaps as 1's; (b) is an output image by the proposed HDR stitching algorithm; as well as (c) and (d) are two output images by using an existing LDR panoramic imaging algorithm.
• Figure 2: Two examples in the proposed panoramic HDR stitching dataset. (a) are images with high exposures. (b) are images with middle exposures. (c) are images with low exposures.
• Figure 3: The diagram of our panoramic HDR imaging. Three panoramic LDR images with different exposures are generated from three differently exposed LDR images with different orientations via a physics-driven deep learning framework. They are combined together to generate a panoramic LDR image with enriched information.
• Figure 4: Generation of $Z^{1\to 3}$ (i.e., $i=1$, $j=3$) via the proposed neural augmentation framework. $Z_m^{1\to 3}$ is produced by using a physics-driven approach and is then refined via a data-driven approach to obtain $Z^{1\to 3}$. Our MEAN is composed of EAGB and SRRG. Our MEAN is on top of the CNN in CycleISP. The multi-scale structure can preserve the high-frequency information. The EAGB can help restore the saturated areas more effectively.
• Figure 5: Diagram of the proposed SRRG. Our SRRG is obtained by adding short-skip connections to the RRG in CycleISP.