LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation
Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, Fatih Porikli
TL;DR
LoRA-X tackles the data-access bottleneck in parameter-efficient fine-tuning by enabling training-free cross-model transfer of LoRA adapters. It constrains weight changes to the source base model subspace, projecting onto target subspaces guided by subspace similarity and an optimal-transport–based transferability cost. Experiments on text-to-image diffusion models (e.g., SD-v1.5, SDXL, and variants) show training-free transfers achieve performance close to trained counterparts across multiple targets, with efficient computation and reduced data dependency. This approach extends the longevity of PEFT adapters and facilitates seamless model evolution in privacy- or license-constrained settings.
Abstract
The rising popularity of large foundation models has led to a heightened demand for parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), which offer performance comparable to full model fine-tuning while requiring only a few additional parameters tailored to the specific base model. When such base models are deprecated and replaced, all associated LoRA modules must be retrained, requiring access to either the original training data or a substantial amount of synthetic data that mirrors the original distribution. However, the original data is often inaccessible due to privacy or licensing issues, and generating synthetic data may be impractical and insufficiently representative. These factors complicate the fine-tuning process considerably. To address this challenge, we introduce a new adapter, Cross-Model Low-Rank Adaptation (LoRA-X), which enables the training-free transfer of LoRA parameters across source and target models, eliminating the need for original or synthetic training data. Our approach imposes the adapter to operate within the subspace of the source base model. This constraint is necessary because our prior knowledge of the target model is limited to its weights, and the criteria for ensuring the adapter's transferability are restricted to the target base model's weights and subspace. To facilitate the transfer of LoRA parameters of the source model to a target model, we employ the adapter only in the layers of the target model that exhibit an acceptable level of subspace similarity. Our extensive experiments demonstrate the effectiveness of LoRA-X for text-to-image generation, including Stable Diffusion v1.5 and Stable Diffusion XL.