Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation

Shiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang, Ziqiang Cui, Dugang Liu, Yuhua Li, Xiuqiang He, Ruixuan Li

TL;DR

TopLoRA advances parameter-efficient fine-tuning by introducing token-wise input-output projections—via a token-dependent diagonal matrix Σ_X—into the LoRA framework. It preserves the original low-rank structure while enabling token-specific adaptations, implemented through a gating network that produces Σ_X and a RMSNorm-based normalization with an exponential transform. Empirically, TopLoRA yields consistent accuracy gains across NLU and NLG tasks and various model scales, often surpassing higher-rank LoRA with substantially fewer parameters. This approach offers a practical route to finer-grained adaptation in large language models, with potential extensions to other domains and further cost-optimization work to mitigate inference overhead.

Abstract

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank. In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection. This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens. To address this limitation, we propose Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner. Formally, the weights of TopLoRA can be expressed as $BΣ_X A$, where $A$ and $B$ are low-rank matrices (as in standard LoRA), and $Σ_X$ is a diagonal matrix generated from each input token $X$. Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections). Extensive experiments across multiple models and datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. The code is available at https://github.com/Leopold1423/toplora-neurips25.

Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation

TL;DR

TopLoRA advances parameter-efficient fine-tuning by introducing token-wise input-output projections—via a token-dependent diagonal matrix Σ_X—into the LoRA framework. It preserves the original low-rank structure while enabling token-specific adaptations, implemented through a gating network that produces Σ_X and a RMSNorm-based normalization with an exponential transform. Empirically, TopLoRA yields consistent accuracy gains across NLU and NLG tasks and various model scales, often surpassing higher-rank LoRA with substantially fewer parameters. This approach offers a practical route to finer-grained adaptation in large language models, with potential extensions to other domains and further cost-optimization work to mitigate inference overhead.

Abstract

Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models (LLMs). LoRA essentially describes the projection of an input space into a low-dimensional output space, with the dimensionality determined by the LoRA rank. In standard LoRA, all input tokens share the same weights and undergo an identical input-output projection. This limits LoRA's ability to capture token-specific information due to the inherent semantic differences among tokens. To address this limitation, we propose Token-wise Projected Low-Rank Adaptation (TopLoRA), which dynamically adjusts LoRA weights according to the input token, thereby learning token-wise input-output projections in an end-to-end manner. Formally, the weights of TopLoRA can be expressed as , where and are low-rank matrices (as in standard LoRA), and is a diagonal matrix generated from each input token . Notably, TopLoRA does not increase the rank of LoRA weights but achieves more granular adaptation by learning token-wise LoRA weights (i.e., token-wise input-output projections). Extensive experiments across multiple models and datasets demonstrate that TopLoRA consistently outperforms LoRA and its variants. The code is available at https://github.com/Leopold1423/toplora-neurips25.
Paper Structure (32 sections, 8 equations, 2 figures, 8 tables)

This paper contains 32 sections, 8 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Motivations
  3. Low-Rank Adaptation
  4. Input-Output Projections of Low-Rank Adaptation
  5. Methodology
  6. Token-wise Projected Low-Rank Adaptation
  7. Optimization of Token-wise Diagonal Matrix
  8. Comparison with Low-Rank Adaptation
  9. Experiments
  10. General Settings
  11. Datasets and Models.
  12. Baseline Methods.
  13. General Hyperparammeters.
  14. Natural Language Understanding Tasks
  15. Implementation Details.
  16. ...and 17 more sections

Figures (2)

  • Figure 1: An illustration of TopLoRA in comparison to LoRA lora. TopLoRA additionally learns a projector $\mathit{\Theta}$ to generate a diagonal matrix $\Sigma_X$ based on the input token $X$, which is then used to adjust the LoRA weights (i.e., the input-output projections) for each token. The operators $\text{Exp}(\cdot)$ and $\text{RMSNorm}(\cdot)$ refer to the exponential function and root mean square normalization rmsnorm, respectively.
  • Figure 2: The scalability analysis on mathematical reasoning tasks using LLama-3-8B. (a) Accuracy of TopLoRA at varying ranks. (b) Accuracy of TopLoRA at different tuning granularity. The figures present only the average accuracy, and the detailed results are available in Appendix \ref{['sec:appendix_result']}.