FPANet: Frequency-based Video Demoireing using Frame-level Post Alignment

TL;DR

FPANet addresses image-video demoireing by learning filters in both frequency and spatial domains and by leveraging three-frame inputs with a frame-level Post Align Module to ensure temporal consistency. The core ideas are the Frequency Selection Fusion (FSF) block, composed of the Frequency Selection Module (FSM) and Cross Scale Fusion Module (CSFM), and the Post Align Module (PAM) for robust temporal alignment. The model optimizes with a multi-term loss that includes spatial, perceptual, and frequency-domain components, and it demonstrates superior performance on the VDmoire dataset in both image and video metrics, as well as strong qualitative restoration. The results indicate that explicit handling of amplitude and phase in the frequency domain, combined with multi-scale spatial features and frame-aligned temporal cues, yields cleaner moiré removal with preserved details and color fidelity, suggesting practical impact for high-quality video demoiréing in real-world pipelines.

Abstract

Moire patterns, created by the interference between overlapping grid patterns in the pixel space, degrade the visual quality of images and videos. Therefore, removing such patterns~(demoireing) is crucial, yet remains a challenge due to their complexities in sizes and distortions. Conventional methods mainly tackle this task by only exploiting the spatial domain of the input images, limiting their capabilities in removing large-scale moire patterns. Therefore, this work proposes FPANet, an image-video demoireing network that learns filters in both frequency and spatial domains, improving the restoration quality by removing various sizes of moire patterns. To further enhance, our model takes multiple consecutive frames, learning to extract frame-invariant content features and outputting better quality temporally consistent images. We demonstrate the effectiveness of our proposed method with a publicly available large-scale dataset, observing that ours outperforms the state-of-the-art approaches in terms of image and video quality metrics and visual experience.

Figures (20)

• Figure 1: Examples of video frames or images that have moiré patterns as visual artifacts. Note that (a) and (b) are consecutive frames, which are extracted from the publicly available VDmoire dataset, while (c) is from the TIP2018 dataset. Target images are shown in the first row, and images with moiré patterns are shown at the bottom. For better visualization, we also provide magnified patches.
• Figure 2: Visualization on the effect of amplitude and phase component over moiré patterns. The orange box generates a synthetic image combined with moiré image amplitude and clean image phase. The green box generates a synthetic image combined with the moiré image phase and clean image amplitude.
• Figure 3: Example for misalignment caused by moiré patterns. Aligned features between reference frame (Frame 2) and target frame (Frame 3) with or without moiré patterns are listed in the first row. To measure the accuracy for alignment, we calculate optical flow using PWC-Net sun2018pwc, illustrating the relative motion by the color coding that indicates the motion vectors (direction and magnitude) with color intensity.
• Figure 4: An overview of our proposed FPANet. FPANet is based on an encoder-decoder architecture design. Frequency Selection Fusion (FSF) block is the core part of FPANet that is responsible for removing moiré patterns using frequency domain information. Also, Post Align Module (PAM) is used for leveraging temporal information between nearby frames.
• Figure 5: Example for moiré pattern caused by the superposition of two gratings and their spectrum. Red and blue dot indicate the moiré impulse which is not appeared in the existing spectrum in the frequency domain