Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks

TL;DR

HyperFormer++ introduces shared hypernetworks that generate task- and layer-conditioned adapters and layer-normalization parameters for every task within a single multi-task transformer. By training only the hypernetworks, task embeddings, and conditional LN, while freezing the backbone, it achieves cross-task knowledge sharing with minimal per-task parameters. Empirically, it delivers strong GLUE performance versus adapters and full fine-tuning, and shows robust few-shot domain transfer and low-resource fine-tuning benefits. The approach demonstrates scalable, parameter-efficient multi-task learning for encoder-decoder models, with released code enabling reproducibility and further development.

Abstract

State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only 0.29% parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.

Figures (3)

• Figure 1: Left: Adapter integration in the T5 model. Right: Our HyperFormer adapter architecture. Following houlsby2019parameter, we include adapter modules after the two feed-forward layers. The Adapter hypernetwork $h_{A}^l$ produces the weights ($\bm{U_{\tau}^l}$ and $\bm{D_{\tau}^l}$) for task-specific adapter modules conditioned on an input task embedding $\bm{I_{\tau}}$. Similarly, the layer normalization hypernetwork $h_{LN}^l$ generates the conditional layer normalization parameters ($\bm{\beta_{\tau}}$ and $\bm{\gamma_{\tau}}$). During training, we only update layer normalizations in T5, hypernetworks, and task embeddings. The compact HyperFormer++ shares the same hypernetworks across all layers and tasks and computes the task embedding based on task, layer id, and position of the adapter module (§\ref{['sec:compact_model']}).
• Figure 2: Results on GLUE for the various number of training samples per task $(100, 500, 1000, 2000, 4000)$. We show mean and standard deviation across 5 seeds.
• Figure 3: Visualization of learned task embeddings by HyperFormer++ BASE.