Enhancing Parameter-Efficient Fine-Tuning of Vision Transformers through Frequency-Based Adaptation

TL;DR

The paper tackles the limitation of parameter-efficient fine-tuning (PEFT) for Vision Transformers (ViTs) by addressing the underrepresentation of high-frequency image patterns. It introduces FreqFit, a simple frequency-domain module inserted between ViT blocks that uses a 2D FFT, a learnable spectral filter, an inverse FFT, and a residual scaling/shifting, compatible with existing PEFT methods. Theoretical support via two theorems shows that FreqFit adds transformations not achievable by spatial-domain PEFT alone and that combining the two is beneficial; extensive experiments across 24 datasets with supervised and self-supervised pre-trained models demonstrate consistent improvements (1%–16%), including notable gains on CIFAR100 and cases outperforming full fine-tuning. The work establishes frequency-domain adaptation as a practical, broadly applicable enhancement for ViT-based foundation models and provides open-source code to facilitate reproducibility.

Abstract

Adapting vision transformer foundation models through parameter-efficient fine-tuning (PEFT) methods has become increasingly popular. These methods optimize a limited subset of parameters, enabling efficient adaptation without the need to fine-tune the entire model while still achieving competitive performance. However, traditional PEFT methods may limit the model's capacity to capture complex patterns, especially those associated with high-frequency spectra. This limitation becomes particularly problematic as existing research indicates that high-frequency features are crucial for distinguishing subtle image structures. To address this issue, we introduce FreqFit, a novel Frequency Fine-tuning module between ViT blocks to enhance model adaptability. FreqFit is simple yet surprisingly effective, and can be integrated with all existing PEFT methods to boost their performance. By manipulating features in the frequency domain, our approach allows models to capture subtle patterns more effectively. Extensive experiments on 24 datasets, using both supervised and self-supervised foundational models with various state-of-the-art PEFT methods, reveal that FreqFit consistently improves performance over the original PEFT methods with performance gains ranging from 1% to 16%. For instance, FreqFit-LoRA surpasses the performances of state-of-the-art baselines on CIFAR100 by more than 10% even without applying regularization or strong augmentation. For reproducibility purposes, the source code is available at https://github.com/tsly123/FreqFiT.

Figures (16)

• Figure 1: Overview of FreqFit
• Figure 2: Performance gains with (left) Imagenet-21K and (right) MoCo
• Figure 4: Relative log amplitudes of Fourier transformed feature maps. $\Delta$ Log amplitude means relative logarithmic amplitude concerning the logarithmic amplitude at normalized frequency $0\pi$ (center) and $1\pi$ (boundary). The brighter the color, the deeper the layer. Our FreqFiT has higher amplitudes compared to the original methods. These visualizations suggest our FreqFit is better at capturing high-frequency components, and potentially leading to better performance. More in Supplementary Material.
• Figure 5: Filters in the frequency domain in different fine-tuning settings. For each case, we show 12 filters that are inserted before the $1^{th}$ to $12^{th}$ blocks of ViTs, which are labeled according to "pre-trained model/incorporated method/task". The center pixel represents the zero frequency and the brighter pixel indicates higher amplitudes. Our visualization shows that the incorporated FreqFiT-VPT, FreqFiT-Adapter, and FreqFit-LoRA can capture high-frequency components by adopting our FreqFiT. Whereas, the FreqFiT-Bias does not show a clear pattern of capturing high-frequency components.
• Figure 6: Performances when applied FreqFit outperform when applied Scale-Shift on 19 tasks of VTab-1K with Imagenet-21K pre-trained model.