High-resolution Photo Enhancement in Real-time: A Laplacian Pyramid Network

TL;DR

This work introduces LLF-LUT++, a real-time high-resolution photo enhancement framework that unifies global tone manipulation and local edge-detail preservation. It combines an image-adaptive 3D-LUT fusion with a spatial-frequency transformer-based weight predictor and a learnable local Laplacian filter operating within an adaptive Laplacian pyramid. The model demonstrates state-of-the-art PSNR/SSIM/LPIPS on HDR+ and MIT FiveK benchmarks, while processing 4K images at roughly 13 ms on a single GPU. This hybrid global-local approach delivers high-quality, edge-preserving enhancements suitable for edge devices and high-resolution workflows. The method also includes extensive ablations and discusses limitations, including potential video-temporal artifacts and biases in learned retouching.

Abstract

Photo enhancement plays a crucial role in augmenting the visual aesthetics of a photograph. In recent years, photo enhancement methods have either focused on enhancement performance, producing powerful models that cannot be deployed on edge devices, or prioritized computational efficiency, resulting in inadequate performance for real-world applications. To this end, this paper introduces a pyramid network called LLF-LUT++, which integrates global and local operators through closed-form Laplacian pyramid decomposition and reconstruction. This approach enables fast processing of high-resolution images while also achieving excellent performance. Specifically, we utilize an image-adaptive 3D LUT that capitalizes on the global tonal characteristics of downsampled images, while incorporating two distinct weight fusion strategies to achieve coarse global image enhancement. To implement this strategy, we designed a spatial-frequency transformer weight predictor that effectively extracts the desired distinct weights by leveraging frequency features. Additionally, we apply local Laplacian filters to adaptively refine edge details in high-frequency components. After meticulously redesigning the network structure and transformer model, LLF-LUT++ not only achieves a 2.64 dB improvement in PSNR on the HDR+ dataset, but also further reduces runtime, with 4K resolution images processed in just 13 ms on a single GPU. Extensive experimental results on two benchmark datasets further show that the proposed approach performs favorably compared to state-of-the-art methods. The source code will be made publicly available at https://github.com/fengzhang427/LLF-LUT.

Figures (10)

• Figure 1: The framework of LLF-LUT++. For global enhancement, the LR image $I_{LR}$ is first fed into a lightweight transformer weight predictor to predict weight points and weight maps for refining HR image $I$ and LR image $I_{LR}$, respectively. Then the coarse refined HR image $\overline{I}$ is decomposed into a Laplacian pyramid. To adaptively refine the high-frequency components, we progressively learn an image-adaptive local Laplacian filter (LLF) based on both high- and low-frequency images. Then, we perform the remapping function of the local Laplacian filter to refine the high-frequency components while preserving the pyramid reconstruction capability. For the level $n-1$, we concatenate the component with the refined LR image $\overline{I}_{LR}$ and its edge map to mitigate potential halo artifacts.
• Figure 2: Illustration of the basis 3D LUTs fusion strategy. (a) present the multiple pixel mapping relationships of an image pair; (b) is the conventional basis 3D LUTs fusion strategy; (c) is the pixel-level basis 3D LUTs fusion strategy.
• Figure 3: The architecture of the proposed spatial-frequency transformer network for weight prediction. The features extracted from input LR images $I_{LR}$ are processed by four encoders to build dictionaries that will be used as inputs for the decoders. The last encoder block will output learnable weight points through a linear layer, while the last decoder block will output learnable weight maps through two convolutional layers. The Fourier-Swin transformer encoder and Swin transformer decoder blocks are illustrated with more details in Fig. \ref{['fig:enc-dec']}.
• Figure 4: (a) is the Fourier-Swin Transformer encoder block; (b) is the basic Swin Transformer decoder block. Here, LN stands for Layer Normalization, W-MSA is Windowed Multi-Head Self-Attention, and W-MCA is Windowed Multi-Head Cross-Attention. FFT1d and inv FFT1d indicate the Fast Fourier Transformer and its inverse.
• Figure 5: Visual comparison with state-of-the-art methods on a 480p resolution test image from the HDR+ dataset hasinoff2016burst. The error maps in the upper left corner facilitate a more precise determination of performance differences. Best viewed in color and by zooming in.