Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning

Bartosz Trojan, Filip Gębala

Abstract

Modern Transformer-based models frequently suffer from miscalibration, producing overconfident predictions that do not reflect true empirical frequencies. This work investigates the calibration dynamics of LoRA: Low-Rank Adaptation and a novel hyper-network-based adaptation framework as parameter-efficient alternatives to full fine-tuning for RoBERTa. Evaluating across the GLUE benchmark, we demonstrate that LoRA-based adaptation consistently achieves calibration parity with (and in specific tasks exceeds) full fine-tuning, while maintaining significantly higher parameter efficiency. We further explore a dynamic approach where a shared hyper-network generates LoRA factors (A and B matrices) to induce structural coupling across layers. This approach produced results similar to standard LoRA fine-tuning, even achieving better MCC on CoLA dataset. Our study also reveal a critical trade-off: constraining the adaptation space (e.g., freezing matrices A) acts as a powerful regularizer that enhances Expected Calibration Error (ECE), but necessitates a carefully balanced sacrifice in downstream task accuracy. To support future research, we provide a unified and reproducible implementation of contemporary calibration metrics, including ECE, MCE, and ACE. Our findings clarify the relationship between parameter efficiency and probabilistic reliability, positioning structured low-rank updates as a viable foundation for uncertainty-aware Transformer architectures. Code available at: https://github.com/btrojan-official/HypeLoRA

HypeLoRA: Hyper-Network-Generated LoRA Adapters for Calibrated Language Model Fine-Tuning

Abstract

Modern Transformer-based models frequently suffer from miscalibration, producing overconfident predictions that do not reflect true empirical frequencies. This work investigates the calibration dynamics of LoRA: Low-Rank Adaptation and a novel hyper-network-based adaptation framework as parameter-efficient alternatives to full fine-tuning for RoBERTa. Evaluating across the GLUE benchmark, we demonstrate that LoRA-based adaptation consistently achieves calibration parity with (and in specific tasks exceeds) full fine-tuning, while maintaining significantly higher parameter efficiency. We further explore a dynamic approach where a shared hyper-network generates LoRA factors (A and B matrices) to induce structural coupling across layers. This approach produced results similar to standard LoRA fine-tuning, even achieving better MCC on CoLA dataset. Our study also reveal a critical trade-off: constraining the adaptation space (e.g., freezing matrices A) acts as a powerful regularizer that enhances Expected Calibration Error (ECE), but necessitates a carefully balanced sacrifice in downstream task accuracy. To support future research, we provide a unified and reproducible implementation of contemporary calibration metrics, including ECE, MCE, and ACE. Our findings clarify the relationship between parameter efficiency and probabilistic reliability, positioning structured low-rank updates as a viable foundation for uncertainty-aware Transformer architectures. Code available at: https://github.com/btrojan-official/HypeLoRA
Paper Structure (17 sections, 4 equations, 2 figures, 2 tables)

This paper contains 17 sections, 4 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Approach
  4. Problem Definition
  5. Layer-Conditioned Low-Rank Updates with Hyper-Network
  6. Training and Inference
  7. Experimental Setup
  8. Datasets
  9. Training and Evaluation Setup
  10. Hyper-network Architectures
  11. Metrics
  12. Results
  13. Calibration of the Existing Methods
  14. Our Approach - Calibration with Hyper-Network
  15. Conclusion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: A hyper-network generates the weights for the Query and Value matrices in all attention blocks of the RoBERTa model, while the original pretrained weights remain frozen. The figure illustrates the approach in which the hyper-network produces both the $A$ and $B$ matrices. In this work, we also present a variant where only the $B$ matrices are generated by the hyper-network, and $A$ matrices are fixed with randomly initialized values, which isn't shown on this figure.
  • Figure 2: Evaluation results on CoLA and SST-2 benchmarks, reported as Matthews Correlation Coefficient and accuracy (top row) alongside Expected Calibration Error (bottom row), averaged across 3 independent random seeds. $A_\text{gen}$ means both matrices $A$ and $B$ are generated, and $A_\text{fix}$ means matrices $A$ are fixed. LoRA hu2022lowrank is included as a baseline. Fixing matrix $A$ improves model calibration, albeit at the cost of task performance across both datasets.