Activation Manifold Projection: Liberating Task-Specific Behaviors from LLM Architectures

TL;DR

The paper tackles architectural lock-in in PEFT by proposing CAST, a direct activation-space transfer framework that learns a mapping between activation manifolds of different LLMs. CAST treats a LoRA as a frozen behavioral kernel and attaches lightweight projection heads per LoRA-adapted layer to translate activations into the source space, apply the frozen kernel, and project back, enabling zero-shot transfer without task data. Using a dual objective that combines $\mathcal{L}_{KL}$ and $\mathcal{L}_{MSE}$ trained on a general corpus, CAST achieves 85–95% of the performance of fully retrained adapters across heterogeneous model pairs, outperforming weight-space transfer baselines like Cross-LoRA. This approach establishes a new state-of-the-art in model interoperability and offers a general, data-free path to portably deploy LoRA adapters across architectures. The work has practical impact by expanding the usable LoRA ecosystem and reducing the cost of adapting models to new architectures while preserving performance.

Abstract

The proliferation of Large Language Model (LLM) architectures presents a fundamental challenge: valuable, task-specific behaviors learned through fine-tuning methods like Low-Rank Adaptation (LoRA) are effectively trapped within their source model's architecture, herein referred to architectural lock-in. Existing transfer methods attempt to bridge this gap by aligning the static weight spaces of models, a brittle and indirect approach that relies on tenuous correlations between parameter geometries. This paper introduces a fundamentally different and more direct paradigm: the Cartridge Activation Space Transfer (CAST), a novel framework that liberates LoRA-encoded behaviors by learning a direct, nonlinear mapping between the activation manifolds, the geometric structures formed by the model's internal neuron activations, of two distinct LLM architectures. CAST treats a pre-trained LoRA as a frozen "behavioral kernel." It learns a set of lightweight, bidirectional projection heads that translate the target model's activation stream into the source model's latent space, apply the frozen kernel, and project the result back. This process, trained on a general text corpus without any task-specific data, effectively decouples the learned skill from the source architecture. We demonstrate that CAST enables true "zero-shot" translation of any standard LoRA adapter. Our experiments, including transfers between heterogeneous model families like Llama-2 and Mistral, show that CAST-translated adapters achieve 85-95\% of the performance of a LoRA fully retrained on the target model, quantitatively outperforming current weight-space transfer techniques and establishing a new state-of-the-art in model interoperability.