Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions

TL;DR

GenLoRA tackles the parameter-inefficiency of LoRA by showing that explicit basis vectors contain redundancy that can be captured with lightweight nonlinear generators driven by latent seeds. By mapping latent vectors through RBF-based generators, GenLoRA synthesizes the A and B matrices, boosting effective rank without proportional parameter growth and preserving low-rank structure with bounded gradients. Empirical results across math, commonsense, and code-generation tasks demonstrate consistent accuracy gains and significant parameter savings versus LoRA and its variants, with strong scalability as rank and group size increase. The approach enables near-zero overhead at inference via weight merging and offers a principled, theoretically grounded framework for nonlinear, generative weight updates in PEFT.

Abstract

Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices. However, standard LoRA follows an explicit-rank paradigm, where increasing model capacity requires adding more rows or columns (i.e., basis vectors) to the low-rank matrices, leading to substantial parameter growth. In this paper, we find that these basis vectors exhibit significant parameter redundancy and can be compactly represented by lightweight nonlinear functions. Therefore, we propose Generative Low-Rank Adapter (GenLoRA), which replaces explicit basis vector storage with nonlinear basis vector generation. Specifically, GenLoRA maintains a latent vector for each low-rank matrix and employs a set of lightweight radial basis functions (RBFs) to synthesize the basis vectors. Each RBF requires far fewer parameters than an explicit basis vector, enabling higher parameter efficiency in GenLoRA. Extensive experiments across multiple datasets and architectures show that GenLoRA attains higher effective LoRA ranks under smaller parameter budgets, resulting in superior fine-tuning performance. The code is available at https://anonymous.4open.science/r/GenLoRA-1519.

Figures (4)

• Figure 1: (Upper) The reconstruction result of the first row vector of a pretrained LoRA matrix $B$. The trajectories in hue illustrate the overlap between the Radial Basis Function (RBF) approximation and the original values. (Lower) Accuracy--parameter trade-off on mathematical reasoning tasks with LLaMA3-8B. The five points for each method correspond to ranks $r = \{2, 4, 8, 16, 32\}$.
• Figure 2: The overall architecture of Generative Low-Rank Adapters (GenLoRA). (a) Explicit Rank: Standard LoRA relies on the explicit-rank paradigm, where model capacity is constrained by the linear dimension of basis vectors. (b) Nonlinearity as Rank: Our proposed paradigm synthesizes basis vectors from latent vectors via generative functions, effectively reducing the parameter cost. (c) RBF-based Generators: The detailed internal workflow of the generator. High-dimensional latent vectors undergo group-wise decomposition and instance-wise normalization to ensure stability and effectiveness. The final basis vectors are synthesized by aggregating weighted Gaussian Radial Basis Functions combined with a linear residual term.
• Figure 3: Accuracy of GenLoRA at varying ranks and group sizes.
• Figure 4: Singular value analysis of Query layers. (Left) Number of singular values $> 0.005$. (Right)) Sum of squared singular values. In the left plot, LoRA and GenLoRA with $r=\{8, 32\}$ overlap, resulting in only the GenLoRA colors being visible.

Theorems & Definitions (11)

• Proposition 3.1: Rank Boundedness
• Proposition 3.2: Gradient Boundedness
• Lemma 1.1
• proof
• Lemma 1.2
• proof
• Lemma 2.1
• proof
• Lemma 2.2
• proof