Frequency Switching Mechanism for Parameter-E!cient Multi-Task Learning

Abstract

Multi-task learning (MTL) aims to enable a single model to solve multiple tasks efficiently; however, current parameter-efficient fine-tuning (PEFT) methods remain largely limited to single-task adaptation. We introduce \textbf{Free Sinewich}, a parameter-efficient multi-task learning framework that enables near-zero-cost weight modulation via frequency switching (\textbf{Free}). Specifically, a \textbf{Sine-AWB (Sinewich)} layer combines low-rank factors and convolutional priors into a single kernel, which is then modulated elementwise by a sinusoidal transformation to produce task-specialized weights. A lightweight Clock Net is introduced to produce bounded frequencies that stabilize this modulation during training. Theoretically, sine modulation enhances the rank of low-rank adapters, while frequency separation decorrelates the weights of different tasks. On dense prediction benchmarks, Free Sinewich achieves state-of-the-art performance-efficiency trade-offs (e.g., up to +5.39\% improvement over single-task fine-tuning with only 6.53M trainable parameters), offering a compact and scalable paradigm based on frequency-based parameter sharing. Project page: \href{https://casperliuliuliu.github.io/projects/Free-Sinewich/}{https://casperliuliuliu.github.io/projects/Free-Sinewich}.

Figures (6)

• Figure 1: Overall comparison of different PEFT-MTL methods. $r$ denotes the rank for the low-rank decomposition modules.
• Figure 2: Illustration of frequency switching. (a) The base matrix space $X$ represents a shared parameter substrate reused across tasks. Each task-specific matrix space $\mathcal{M}_t$ denotes the domain where the transformed matrix resides. Different sine transformations with task-dependent frequencies $\omega_t$ map the same base matrix into $M_t$, producing task-specific matrices with distinct properties. (b) Each $\omega_t$ defines a unique sine wave, inducing a nonlinear mapping $\mathcal{F}_{\omega_t}$ that transforms the shared base matrix into task-specific representations elementwise. This illustrates how varying $\omega_t$ enables task specialization without parameter duplication.
• Figure 3: Overview of the Free Sinewich framework: (a) A Swin Transformer Tiny liu2021Swin serves as the shared encoder. The encoder receives image patch tokens with task tokens prepended to them. Following the VPT-shallow strategy jia2022vpt, the task tokens are introduced only at the first Transformer stage. Task-agnostic features are extracted through the task-agnostic module (TA-Module) in all Transformer blocks except for the last block, while the last block employs a task-specific module (TS-Module) to capture task-dependent representations. (b) Details of the TA-Module, which are basically the same as LoRA. (c) The TS-Module includes a lightweight Clock Net (LCN) that takes task token $\boldsymbol{p}_t$ as input and determines a task frequency $\omega_t$. This frequency is used in the sine transformation to enhance the representational power of the low-rank adapter. Conceptually, we switch the shared base matrix $M_{\mathsf{AWB}}$ into different transient, task-specialized matrices $M_t$'s (ghost icon). The transient $M_t$'s are instantiated on-the-fly to extract task-specific features, enabling efficient and scalable PEFT-MTL.
• Figure 4: Visual comparison of semantic segmentation on the Pascal-Context dataset. The Free Sinewich method yields fuller and more coherent segmentation masks, capturing finer-grained boundary details than TADFormer.
• Figure 5: Effect of backbone pretraining and model capacity on Free Sinewich. (a) Comparison between Swin-T pretrained on ImageNet-1K and ImageNet-22K. (b) Performance comparison between Swin-T and Swin-B backbones (both pretrained on ImageNet-22K).