Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Spatial and Frequency Information for Under-Display Camera Image Restoration

Kyusu Ahn, Jinpyo Kim, Chanwoo Park, JiSoo Kim, Jaejin Lee

TL;DR

This work tackles the challenging problem of restoring images captured with under-display cameras, where degradations include noise, blur, reduced transmittance, and especially global flares. It introduces SFIM, a four-level architecture that combines CNN-based spatial processing (level 1) with FFT-based global modeling (levels 2–4), unified by an attention-driven multi-level integration block (AMIB). The approach leverages both local and global information, with a loss that combines spatial and frequency-domain terms to guide restoration. Empirical results on three UDC benchmarks demonstrate state-of-the-art performance, with strong improvements in flare suppression and texture fidelity, highlighting the value of explicitly integrating spatial and frequency information for broad, real-world degradations.

Abstract

Under-Display Camera (UDC) houses a digital camera lens under a display panel. However, UDC introduces complex degradations such as noise, blur, decrease in transmittance, and flare. Despite the remarkable progress, previous research on UDC mainly focuses on eliminating diffraction in the spatial domain and rarely explores its potential in the frequency domain. It is essential to consider both the spatial and frequency domains effectively. For example, degradations, such as noise and blur, can be addressed by local information (e.g., CNN kernels in the spatial domain). At the same time, tackling flares may require leveraging global information (e.g., the frequency domain). In this paper, we revisit the UDC degradations in the Fourier space and figure out intrinsic frequency priors that imply the presence of the flares. Based on this observation, we propose a novel multi-level DNN architecture called SFIM. It efficiently restores UDC-distorted images by integrating local and global (the collective contribution of all points in the image) information. The architecture exploits CNNs to capture local information and FFT-based models to capture global information. SFIM comprises a spatial domain block (SDB), a Frequency Domain Block (FDB), and an Attention-based Multi-level Integration Block (AMIB). Specifically, SDB focuses more on detailed textures such as noise and blur, FDB emphasizes irregular texture loss in extensive areas such as flare, and AMIB enables effective cross-domain interaction. SFIM's superior performance over state-of-the-art approaches is demonstrated through rigorous quantitative and qualitative assessments across three UDC benchmarks.

Integrating Spatial and Frequency Information for Under-Display Camera Image Restoration

TL;DR

This work tackles the challenging problem of restoring images captured with under-display cameras, where degradations include noise, blur, reduced transmittance, and especially global flares. It introduces SFIM, a four-level architecture that combines CNN-based spatial processing (level 1) with FFT-based global modeling (levels 2–4), unified by an attention-driven multi-level integration block (AMIB). The approach leverages both local and global information, with a loss that combines spatial and frequency-domain terms to guide restoration. Empirical results on three UDC benchmarks demonstrate state-of-the-art performance, with strong improvements in flare suppression and texture fidelity, highlighting the value of explicitly integrating spatial and frequency information for broad, real-world degradations.

Abstract

Under-Display Camera (UDC) houses a digital camera lens under a display panel. However, UDC introduces complex degradations such as noise, blur, decrease in transmittance, and flare. Despite the remarkable progress, previous research on UDC mainly focuses on eliminating diffraction in the spatial domain and rarely explores its potential in the frequency domain. It is essential to consider both the spatial and frequency domains effectively. For example, degradations, such as noise and blur, can be addressed by local information (e.g., CNN kernels in the spatial domain). At the same time, tackling flares may require leveraging global information (e.g., the frequency domain). In this paper, we revisit the UDC degradations in the Fourier space and figure out intrinsic frequency priors that imply the presence of the flares. Based on this observation, we propose a novel multi-level DNN architecture called SFIM. It efficiently restores UDC-distorted images by integrating local and global (the collective contribution of all points in the image) information. The architecture exploits CNNs to capture local information and FFT-based models to capture global information. SFIM comprises a spatial domain block (SDB), a Frequency Domain Block (FDB), and an Attention-based Multi-level Integration Block (AMIB). Specifically, SDB focuses more on detailed textures such as noise and blur, FDB emphasizes irregular texture loss in extensive areas such as flare, and AMIB enables effective cross-domain interaction. SFIM's superior performance over state-of-the-art approaches is demonstrated through rigorous quantitative and qualitative assessments across three UDC benchmarks.

Paper Structure

This paper contains 38 sections, 13 equations, 15 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Low-level image restoration tasks.
  4. UDC datasets.
  5. UDC image restoration models.
  6. Architecture of SFIM
  7. Overview
  8. Spatial Domain Blocks
  9. Frequency Domain Blocks
  10. Integrating Spatial-Frequency Information
  11. Loss Functions
  12. Experiments
  13. Experimental Settings
  14. Comparison with Other Models
  15. Quantitative comparison.
  16. ...and 23 more sections

Figures (15)

  • Figure 1: Spatial and frequency analysis for the various degradations, such as (a) gaussian noise huang2015single, (b) blur nah2017deep, (c) lens flare dai2022flare7k, (d) T-OLED zhou2021image, (e) P-OLED zhou2021image, (f) SYNTH feng2021removing, and (g) UDC-SIT ahn2024udc. The existing UDC datasets span from (d) to (g). In the second row, pixel value differences between degraded and ground-truth images are represented in the spatial domain. In (a), pixel values are magnified ten times to improve noise visualization. The third row shows the frequency domain spectral amplitude differences between the degraded and ground-truth images.
  • Figure 2: Overall architecture of SFIM.
  • Figure 3: The structure of the SDB that consists of eight RDBs.
  • Figure 4: The structure of the FDB that consists of an FSAS and a DFFN.
  • Figure 5: The structure of spatial and frequency integration modules. (a) The proposed attention-based multi-level integration block (AMIB). (b) SAM. (c) FAM.
  • ...and 10 more figures