The Devil is in the Frequency: Geminated Gestalt Autoencoder for Self-Supervised Visual Pre-Training

TL;DR

This work rethinks Masked Image Modeling by introducing Geminated Gestalt AutoEncoder (Ge$^2$-AE), which jointly reconstructs content in pixel and Fourier frequency spaces to capture both local details and global semantics. A lightweight Frequency Decoder with a Fourier Spectrum Perceiver enforces informative frequency emphasis and interacts with the pixel branch via reciprocal constraints, alleviating over-smoothing observed in pixel-only targets. Empirically, Ge$^2$-AE yields robust, transferable representations, showing competitive ImageNet-1K performance and strong transfer on COCO and ADE20K, while also providing insights through power-law and CKA analyses. This frequency-domain perspective offers a practical and scalable pathway to enhance self-supervised visual pre-training without extra data or heavy tokenizer frameworks.

Abstract

The self-supervised Masked Image Modeling (MIM) schema, following "mask-and-reconstruct" pipeline of recovering contents from masked image, has recently captured the increasing interest in the multimedia community, owing to the excellent ability of learning visual representation from unlabeled data. Aiming at learning representations with high semantics abstracted, a group of works attempts to reconstruct non-semantic pixels with large-ratio masking strategy, which may suffer from "over-smoothing" problem, while others directly infuse semantics into targets in off-line way requiring extra data. Different from them, we shift the perspective to the Fourier domain which naturally has global perspective and present a new Masked Image Modeling (MIM), termed Geminated Gestalt Autoencoder (Ge$^2$-AE) for visual pre-training. Specifically, we equip our model with geminated decoders in charge of reconstructing image contents from both pixel and frequency space, where each other serves as not only the complementation but also the reciprocal constraints. Through this way, more robust representations can be learned in the pre-trained encoders, of which the effectiveness is confirmed by the juxtaposing experimental results on downstream recognition tasks. We also conduct several quantitative and qualitative experiments to investigate the learning behavior of our method. To our best knowledge, this is the first MIM work to solve the visual pre-training through the lens of frequency domain.

Figures (10)

• Figure 1: Demonstrations of reconstructed images (1st. row), Fourier spectrum maps (2nd. row) and phase-only images (3rd. row) yielded by MAE and our proposed method. The (a) column is the raw image and corresponding maps, while (b) and (c) are results of MAE regarding pixel and frequency as targets. (d) are ours. The direct predictions are highlighted by red boundaries, and their corresponding pixel or frequency maps are obtained by 2D-FFT or 2D-IFFT. The phase-only images indicating semantics are obtained by setting the amplitude component to a constant.
• Figure 2: Architecture of the Proposed Geminated Gestalt Autoencoder ($\text{Ge}^2$-AE). The encoder receives the unmasked patches to yield visible tokens, which are sent to the geminated structure decoders together with masked tokens to recover in pixel and Fourier domain constrained by each other. Best viewed in color and zoomed in.
• Figure 3: Power Law curves and CKA similarity curves of MAE and our $\text{Ge}^2$-AE after pre-trained and fine-tuned. All experiments are performed on IN1K validation set with ViT base model adopted as encoder.
• Figure 4: CKA similarities between all pairs of layers across MAE and our $\text{Ge}^2$-AE trained on IN1K. The horizontal axes of heatmaps and vertical axes indexing the layers from input to output. Best viewed in color and zoom in.
• Figure 5: Visualizations of the predicted results from MAE and our Geminated Gestalt Autoencoder ($\text{Ge}^2$-AE) pre-trained on IN1K dataset. "PD." and "FD." are short for pixel decoder and frequency decoder respectively. Their outputs are $\mathbf{P}$ and $\mathbf{\tilde{Q}}$, corresponding to the counterparts in Fig. \ref{['fig:arch']}(a). Our method can yield results with more necessary global frequency and local details than MAE to overcome the "over-smoothing" issue. Best viewed in color and zoomed in.