A Cycle Ride to HDR: Semantics Aware Self-Supervised Framework for Unpaired LDR-to-HDR Image Reconstruction

TL;DR

CycleHDR tackles HDR reconstruction from LDR using unpaired data by learning bidirectional LDR↔HDR mappings with cycle-consistency. The model integrates semantic awareness through a CLIP-based contrastive loss and a semantic segmentation loss, and uses heuristic-based artifact and exposure saliency to guide training. A novel ConvLSTM-based artifact-aware feedback generator with exposure-aware skip connections is paired with CLIP embeddings, and a six-term loss including adversarial, cycle, identity, contrastive, semantic, and heuristic terms is optimized on tone-mapped HDR/LDR. Experiments on multiple datasets show state-of-the-art HDR reconstruction and competitive LDR reconstructions, including cross-dataset and mixed unpaired settings, demonstrating strong semantic and perceptual fidelity without paired supervision.

Abstract

Reconstruction of High Dynamic Range (HDR) from Low Dynamic Range (LDR) images is an important computer vision task. There is a significant amount of research utilizing both conventional non-learning methods and modern data-driven approaches, focusing on using both single-exposed and multi-exposed LDR for HDR image reconstruction. However, most current state-of-the-art methods require high-quality paired {LDR;HDR} datasets with limited literature use of unpaired datasets, that is, methods that learn the LDR-HDR mapping between domains. This paper proposes CycleHDR, a method that integrates self-supervision into a modified semantic- and cycle-consistent adversarial architecture that utilizes unpaired LDR and HDR datasets for training. Our method introduces novel artifact- and exposure-aware generators to address visual artifact removal. It also puts forward an encoder and loss to address semantic consistency, another under-explored topic. CycleHDR is the first to use semantic and contextual awareness for the LDR-HDR reconstruction task in a self-supervised setup. The method achieves state-of-the-art performance across several benchmark datasets and reconstructs high-quality HDR images. The official website of this work is available at: https://github.com/HrishavBakulBarua/Cycle-HDR

Figures (19)

• Figure 1: Qualitative comparison of the proposed CycleHDR and state-of-the-art SelfHDR SelfHDR methods. CycleHDR handles the overexposed portions in the sky more realistically. Here, a single image is divided into four parts, each rendered using a different approach.
• Figure 2: Overview of the proposed CycleHDR architecture (see \ref{['subsec:architecture_modules']} and \ref{['supsubsec:architecture_modules']}) where $x$ and $y$ represent LDR and HDR images, respectively. The method is trained with six objectives: adversarial, cycle consistency, identity, heuristic-based, contrastive, and semantic segmentation (see \ref{['subsec:loss_functions']} and \ref{['supsubsec:loss_functions']}).
• Figure 3: Examples of HDR images reconstructed with our method and recent state-of-the-art methods.
• Figure 4: Examples of HDR images using the SingleHDR(W) le2023single U-Net with and without our feedback mechanism on images from the DrTMO endo2017deep.
• Figure 5: Ablation results for $\mathcal{L}_{\text{id}}$, $\mathcal{L}_{\text{con}}$, and $\mathcal{L}_{\text{sem}}$ with images from the HDR-Synth & HDR-Real liu2020single and LDR-HDR pair datasets jang2020dynamic.