Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

W2T: LoRA Weights Already Know What They Can Do

Xiaolong Han, Ferrante Neri, Zijian Jiang, Fang Wu, Yanfang Ye, Lu Yin, Zehong Wang

Abstract

Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token.

W2T: LoRA Weights Already Know What They Can Do

Abstract

Each LoRA checkpoint compactly stores task-specific updates in low-rank weight matrices, offering an efficient way to adapt large language models to new tasks and domains. In principle, these weights already encode what the adapter does and how well it performs. In this paper, we ask whether this information can be read directly from the weights, without running the base model or accessing training data. A key obstacle is that a single LoRA update can be factorized in infinitely many ways. Without resolving this ambiguity, models trained on the factors may fit the particular factorization rather than the underlying update. To this end, we propose \methodfull, which maps each LoRA update to a provably canonical form via QR decomposition followed by SVD, so that all equivalent factorizations share the same representation. The resulting components are then tokenized and processed by a Transformer to produce a weight-space embedding. Across language and vision LoRA collections, W2T achieves strong results on attribute classification, performance prediction, and adapter retrieval, demonstrating that LoRA weights reliably indicate model behavior once factorization ambiguity is removed. Code is available at https://github.com/xiaolonghan2000/Weight2Token.
Paper Structure (59 sections, 2 theorems, 21 equations, 8 figures, 9 tables, 1 algorithm)

This paper contains 59 sections, 2 theorems, 21 equations, 8 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Low-rank adaptation.
  4. Weight space learning.
  5. Identifying Canonical Object of LoRA
  6. Canonical object.
  7. Construction from low-rank factors.
  8. W2T: Weight-to-Token Modeling
  9. Framework overview.
  10. Tokenization.
  11. Rank-level modeling.
  12. Position-level modeling.
  13. Experiments
  14. Setup
  15. Experimental Setting.
  16. ...and 44 more sections

Key Result

Proposition 3.1

If $(\mathbf{B}',\mathbf{A}')=(\mathbf{B}\mathbf{G},\,\mathbf{G}^{-1}\mathbf{A})$ for some $\mathbf{G}\in\mathrm{GL}(r)$, then $\mathbf{B}'\mathbf{A}'=\mathbf{B}\mathbf{A}$, and the decomposition in Eq. eq:compact_svd is identical under a fixed sign and ordering convention.

Figures (8)

  • Figure 1: LoRA updates already encode downstream behaviors, but reading them requires a symmetry-aware representation. (a) LoRA checkpoints trained from raw data are treated as data objects for downstream tasks. (b) Symmetry-agnostic processing of raw LoRA factors. (c) Symmetry-aware decomposition of LoRA updates into canonical components.
  • Figure 2: Overview of the proposed W2T modeling framework for LoRA weight space. (a) LoRA factors are transformed into a canonical representation via symmetry-aware QR--SVD decomposition, yielding rank-wise components under a fixed deterministic convention. (b) Each rank-wise component is projected into a rank token through modulated tokenization. (c) Rank tokens within each position are seen as a set and aggregated via $\sigma$-guided pooling. (d) Position tokens are globally modeled to produce a weight-space embedding for downstream tasks.
  • Figure 3: Local retrieval neighborhood for a 8-shot BoolQ query. Each subplot shows the same query (star) amid a shared pool. W2T places relevant adapters (dark) closest to the query with top-1 hit and rel@5=5, while baselines surface more irrelevant neighbors.
  • Figure 4: Ablation on attribute classification. Results are reported on CelebA-LoRA. Removing QR--SVD causes the largest performance drop, while removing rank-level or position-level modeling leads to smaller but consistent degradations.
  • Figure 5: Practical GL($r$) invariance. The horizontal axis is perturbation strength $\alpha$ in random GL($r$) reparameterization (larger $\alpha$ means stronger perturbation). Left: embedding drift under increasing perturbation. Right: prediction consistency under the same perturbation schedule.
  • ...and 3 more figures

Theorems & Definitions (3)

  • Proposition 3.1: $\mathrm{GL}(r)$-invariance
  • Proposition 3.2: QR-based construction
  • Definition A.1: LoRA Reparameterisation Symmetry