Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning

M. H. I. Abdalla, Zhipin Wang, Christian Frey, Steffen Eger, Josif Grabocka

TL;DR

Zhyper introduces a parameter-efficient factorized hypernetwork that conditions LoRA adapters via context-specific modulation signals, enabling dynamic LLM fine-tuning from textual descriptions. For each transformer layer and attention type, a hypernetwork outputs a rank-r vector that modulates the corresponding LoRA matrices through either a diagonal or full-rank context matrix, enabling context-aware adaptation with far fewer parameters. The method achieves competitive accuracy on diverse tasks while reducing trainable parameters by up to 26x compared to strong baselines, and demonstrates superior generalization in cultural alignment across unseen regions and countries. This approach supports more sustainable, fine-grained LLM conditioning by decoupling context encoding from full adapter generation, facilitating efficient deployment in multi-context settings.

Abstract

Large Language Model (LLM) conditioning refers to instructing an LLM to generate content in accordance with the norms and values of a specific culture, beliefs of a particular political orientation, or any desired text-specified semantic conditioning. Unfortunately, prompt engineering does not ensure that LLMs behave in accordance with a desired conditioning due to the inductive bias of the pre-training and alignment datasets. Prior works have focused on fine-tuning LLMs by directly conditioning the LoRA weights; however, such methods introduce a large number of parameters. As a remedy, we propose Zhyper, a parameter-efficient factorized hypernetwork framework that generates context-aware LoRA adapters from textual descriptions. Experiments on multiple benchmarks show that Zhyper achieves competitive performance with up to 26x fewer parameters than the state-of-the-art baselines. Furthermore, we extend Zhyper to cultural alignment, demonstrating improved generalization to out-of-domain settings and a better capturing of fine-grained contextual values.

Zhyper: Factorized Hypernetworks for Conditioned LLM Fine-Tuning

TL;DR

Zhyper introduces a parameter-efficient factorized hypernetwork that conditions LoRA adapters via context-specific modulation signals, enabling dynamic LLM fine-tuning from textual descriptions. For each transformer layer and attention type, a hypernetwork outputs a rank-r vector that modulates the corresponding LoRA matrices through either a diagonal or full-rank context matrix, enabling context-aware adaptation with far fewer parameters. The method achieves competitive accuracy on diverse tasks while reducing trainable parameters by up to 26x compared to strong baselines, and demonstrates superior generalization in cultural alignment across unseen regions and countries. This approach supports more sustainable, fine-grained LLM conditioning by decoupling context encoding from full adapter generation, facilitating efficient deployment in multi-context settings.

Abstract

Large Language Model (LLM) conditioning refers to instructing an LLM to generate content in accordance with the norms and values of a specific culture, beliefs of a particular political orientation, or any desired text-specified semantic conditioning. Unfortunately, prompt engineering does not ensure that LLMs behave in accordance with a desired conditioning due to the inductive bias of the pre-training and alignment datasets. Prior works have focused on fine-tuning LLMs by directly conditioning the LoRA weights; however, such methods introduce a large number of parameters. As a remedy, we propose Zhyper, a parameter-efficient factorized hypernetwork framework that generates context-aware LoRA adapters from textual descriptions. Experiments on multiple benchmarks show that Zhyper achieves competitive performance with up to 26x fewer parameters than the state-of-the-art baselines. Furthermore, we extend Zhyper to cultural alignment, demonstrating improved generalization to out-of-domain settings and a better capturing of fine-grained contextual values.
Paper Structure (28 sections, 6 equations, 5 figures, 11 tables)

This paper contains 28 sections, 6 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Zhyper - Conditioned LLM Tuning
  3. Preliminaries.
  4. Architecture
  5. Complexity Analysis
  6. Experiments
  7. Use case on Task Conditioning
  8. Use case on Cultural Alignment
  9. Related Work
  10. Conclusion
  11. Ethics Statement
  12. Training Procedure And Hyperparameter
  13. Hyperparameter Tuning
  14. Full Task Analysis
  15. Cultural Conditions Generation
  16. ...and 13 more sections

Figures (5)

  • Figure 1: We introduce a novel parameter-efficient architecture for conditioned LLM finetuning based on hypernetwork-conditioned LoRA adapters
  • Figure 2: (Left) Average performance (higher is better); (Right) Performance rank (lower is better). Our method lies in the Pareto front optimality between performance and the number of parameters.
  • Figure 3: Critical Difference (CD) diagrams comparing our method with T2L across LoRA ranks. Lower rank is better. Unconditioned is the base model without any fine-tuning. Groups that are not significantly different are connected by a black bar.
  • Figure 4: Model output based on text conditions. From left to right: unconditioned model, Europe-conditioned model and Germany-conditioned model.
  • Figure 5: Average Performance Rank on the benchmark validation set (lower is better). For MTL, best variant is at $r=8$, T2L, $r=16$ and Zhyper $r=8, diag$.