Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Efficient Post-Training via Fourier-Driven Adapter Architectures

Donggyun Bae, Jongil Park

TL;DR

FAA introduces a Fourier-driven adapter framework for parameter-efficient fine-tuning of large language models by injecting Random Fourier Features into adapters to enable frequency-aware modulation of semantic information. The approach combines a frequency-aware activation, adaptive frequency weights, and sparsity regularization, while freezing the base model to maintain representation capacity. Empirical results across GLUE, E2E NLG, and instruction-tuning benchmarks show FAA consistently matches or surpasses strong PEFT baselines with competitive efficiency. The work demonstrates that spectral processing in adapters yields robust cross-domain generalization and strong generation and reasoning abilities, highlighting the practical value of spectral analysis for efficient LLM adaptation.

Abstract

We propose a novel framework, termed Fourier-Activated Adapter (FAA), for parameter-efficient fine-tuning of large pre-trained language models. By incorporating random Fourier features into lightweight adapter modules, FAA decomposes intermediate representations into complementary low- and high-frequency components, enabling frequency-aware modulation of semantic information. This design allows the model to selectively emphasize informative frequency bands during adaptation while preserving the representational capacity of the frozen backbone. Extensive experiments on GLUE, E2E NLG, and instruction-tuning benchmarks demonstrate that FAA consistently achieves competitive or superior performance compared to existing parameter-efficient fine-tuning methods, while maintaining low computational and memory overhead. Ablation studies further verify the effectiveness of frequency-aware activation and adaptive weighting mechanisms, highlighting FAA as a robust and efficient approach for post-training large language models.

Towards Efficient Post-Training via Fourier-Driven Adapter Architectures

TL;DR

FAA introduces a Fourier-driven adapter framework for parameter-efficient fine-tuning of large language models by injecting Random Fourier Features into adapters to enable frequency-aware modulation of semantic information. The approach combines a frequency-aware activation, adaptive frequency weights, and sparsity regularization, while freezing the base model to maintain representation capacity. Empirical results across GLUE, E2E NLG, and instruction-tuning benchmarks show FAA consistently matches or surpasses strong PEFT baselines with competitive efficiency. The work demonstrates that spectral processing in adapters yields robust cross-domain generalization and strong generation and reasoning abilities, highlighting the practical value of spectral analysis for efficient LLM adaptation.

Abstract

We propose a novel framework, termed Fourier-Activated Adapter (FAA), for parameter-efficient fine-tuning of large pre-trained language models. By incorporating random Fourier features into lightweight adapter modules, FAA decomposes intermediate representations into complementary low- and high-frequency components, enabling frequency-aware modulation of semantic information. This design allows the model to selectively emphasize informative frequency bands during adaptation while preserving the representational capacity of the frozen backbone. Extensive experiments on GLUE, E2E NLG, and instruction-tuning benchmarks demonstrate that FAA consistently achieves competitive or superior performance compared to existing parameter-efficient fine-tuning methods, while maintaining low computational and memory overhead. Ablation studies further verify the effectiveness of frequency-aware activation and adaptive weighting mechanisms, highlighting FAA as a robust and efficient approach for post-training large language models.
Paper Structure (46 sections, 9 equations, 4 figures, 10 tables)

This paper contains 46 sections, 9 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parameter-Efficient Fine-Tuning Techniques
  4. Frequency Domain Enhancement and Fourier Transform
  5. Kolmogorov-Arnold Networks (KAN) and Fourier Activation
  6. Methodology
  7. Fourier Activation Adapter and Frequency Response Enhancement
  8. Random Fourier Feature Transformation and Frequency-Aware Activation Mechanism
  9. Dynamic Enhancement of High-Frequency Features
  10. Adaptive Frequency Weight Adjustment and Training Objective
  11. Adaptive Frequency Weight Adjustment
  12. Sparsity Constraints and Training Objective
  13. FAA Integration with Base Models
  14. Structural Integration and Information Flow Modeling
  15. Parameter Freezing and Fine-Tuning Strategy
  16. ...and 31 more sections

Figures (4)

  • Figure 1: Comparison of traditional adapters and Fourier-activated adapters (FAA). The top shows the traditional adapter, which consists of a feed-forward down-projection, a nonlinearity (e.g., a GELU function), and a feed-forward up-projection. The bottom shows the Fourier-activated adapter (FAA). The fixed activation function is replaced by RFF+GELU, where RFF first controls the consistency of the generated frequency features in the numerical scale through the normalization factor $\sqrt{\frac{2}{D_{\text{rff}}}}$, then controls the frequency distribution through the $\beta$ coefficient, and then combines with GELU through learnable coefficients $a$ and $b$ to achieve a frequency-aware Fourier activation strategy. Note that the adapter layers are interspersed between the attention layer and the feed-forward network.
  • Figure 2: Frequency perception experiment on CoLA (upper) and Wikitext (lower)
  • Figure 3: Frequency perception experiment on AG_NEWS (upper) , MRPC(mid) and SST-2 (lower)
  • Figure 4: Performance of different num_grids values.