Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models

Haomin Qi, Chengbo Huang, Zihan Dai, Yunkai Gao

TL;DR

The paper tackles the challenge of adapting large multilingual models under limited resources by proposing a governance-aware hybrid PEFT approach. It fuses gradient-aligned LoRA updates with structured orthogonal BOFT transforms and adds selective unitary constraints to stabilize deep Transformer stacks, guided by lightweight data governance. Across GLUE, GSM8K, MT-Bench, HumanEval, XNLI, and FLORES, the method achieves close-to-full fine-tuning performance with fewer trainable parameters, improving calibration and cross-language parity while maintaining a favorable cost-quality frontier. These results position the hybrid, unitary-stabilized PEFT as a practical, scalable path for resource-efficient multilingual model adaptation with robust deployment properties.

Abstract

We present a governance-aware hybrid fine-tuning framework for multilingual, low-resource adaptation of large language models. The core algorithm combines gradient-aligned low-rank updates with structured orthogonal transformations through layer-wise mixing and introduces unitary constraints in selected sub-layers to stabilize deep optimization. In tandem with lightweight, label-free data governance steps, including language identification, near-duplicate removal, and quality filtering, the framework targets accuracy, calibration, and cross-language parity under tight compute budgets. Across XNLI and FLORES, the hybrid approach delivers consistent gains over strong PEFT baselines while maintaining directional balance and improving probability calibration, as shown in Tables II and III. It is more resilient to lightweight orthographic variants, as shown in Table IV, and benefits additively from simple governance steps, as shown in Table V. Training footprint measurements indicate modest overhead and a favorable cost-quality frontier, as shown in Table VI and Figure 2. Together, these results show that hybrid and unitary PEFT provide a stable and accessible path to resource-efficient multilingual adaptation when paired with practical data governance.

Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models

TL;DR

The paper tackles the challenge of adapting large multilingual models under limited resources by proposing a governance-aware hybrid PEFT approach. It fuses gradient-aligned LoRA updates with structured orthogonal BOFT transforms and adds selective unitary constraints to stabilize deep Transformer stacks, guided by lightweight data governance. Across GLUE, GSM8K, MT-Bench, HumanEval, XNLI, and FLORES, the method achieves close-to-full fine-tuning performance with fewer trainable parameters, improving calibration and cross-language parity while maintaining a favorable cost-quality frontier. These results position the hybrid, unitary-stabilized PEFT as a practical, scalable path for resource-efficient multilingual model adaptation with robust deployment properties.

Abstract

We present a governance-aware hybrid fine-tuning framework for multilingual, low-resource adaptation of large language models. The core algorithm combines gradient-aligned low-rank updates with structured orthogonal transformations through layer-wise mixing and introduces unitary constraints in selected sub-layers to stabilize deep optimization. In tandem with lightweight, label-free data governance steps, including language identification, near-duplicate removal, and quality filtering, the framework targets accuracy, calibration, and cross-language parity under tight compute budgets. Across XNLI and FLORES, the hybrid approach delivers consistent gains over strong PEFT baselines while maintaining directional balance and improving probability calibration, as shown in Tables II and III. It is more resilient to lightweight orthographic variants, as shown in Table IV, and benefits additively from simple governance steps, as shown in Table V. Training footprint measurements indicate modest overhead and a favorable cost-quality frontier, as shown in Table VI and Figure 2. Together, these results show that hybrid and unitary PEFT provide a stable and accessible path to resource-efficient multilingual adaptation when paired with practical data governance.

Paper Structure

This paper contains 20 sections, 17 equations, 4 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Parameter-Efficient Fine-Tuning (PEFT)
  4. Orthogonal and Unitary Parameterizations
  5. Multilingual Adaptation, Calibration, and Robustness
  6. Data Governance and Budget-Aware Adaptation
  7. Methodology
  8. Low-Rank Adaptation (LoRA)
  9. Butterfly Orthogonal Fine-Tuning (BOFT)
  10. LoRA with Gradient Approximation (LoRA-GA)
  11. Unitary Evolution RNN (uRNN)
  12. Hybrid Fine-Tuning Approach
  13. Experiments
  14. Setup and Evaluation Protocol
  15. General-Purpose Results
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Few-shot scaling on XNLI (BloomZ-7B1). Macro accuracy at 0, 8, 32, and 128 shots.
  • Figure 2: Cost–quality frontier across methods. Upper-left is better.
  • Figure 3: Per-epoch training time and peak GPU memory per method across model scales.
  • Figure 4: Training stability on Wizard-Vicuna-30B across ten epochs: gradient norm and validation loss.