Neural Organ Transplantation (NOT): Checkpoint-Based Modular Adaptation for Transformer Models
Ahmad Al-Zuraiqi
TL;DR
This work addresses the data-sharing and transferability bottlenecks of adapting large language models by proposing Neural Organ Transplantation (NOT), a checkpoint-based modular transfer framework. NOT extracts contiguous middle layers as donor organs, trains them standalone with a minimal wrapper, saves them as self-contained donor checkpoints (donor_layers.pt), and transplant them into compatible recipient models using direct replacement or bridge-mediated insertion, followed by brief recovery fine-tuning. Across three decoder-only architectures (GPT-2 124M, TinyLlama 1.1B, GPT-OSS 20B), NOT yields 2.8–38.6× perplexity improvements over LoRA with 2.3–28.2× faster training, while demonstrating that early insertion positions and middle-layer extraction maximize transferability. The approach enables privacy-preserving expertise sharing, versioned capabilities, and rapid deployment, especially at scale where cross-domain transfer surprisingly improves for the largest model, indicating regularization benefits. Limitations exist for encoder-based models, and future work will explore automatic position selection, cross-architecture transfer, and adaptive organ sizing to broaden applicability.
Abstract
We introduce Neural Organ Transplantation (NOT), a modular adaptation framework that enables trained transformer layers to function as reusable transferable checkpoints for domain adaptation. Unlike conventional fine-tuning approaches that tightly couple trained parameters to specific model instances and training data, NOT extracts contiguous layer subsets ("donor organs") from pre-trained models, trains them independently on domain-specific data, and saves them as standalone checkpoint files that can be transplanted into compatible recipient models without access to the original training data. Through experiments on three decoder-only transformer architectures spanning 124M to 20B parameters (GPT-2, TinyLlama, and GPT-OSS), we demonstrate that donor transplantation substantially outperforms existing adaptation methods, achieving an order-of-magnitude improvement in perplexity over LoRA while training significantly faster. The method exhibits position dependence, with early insertion positions yielding optimal results. Cross-domain transfer at billion-parameter scale reveals unexpected regularization benefits. These findings demonstrate that transformer middle layers can support efficient modular transfer for decoder-only architectures, enabling privacy-preserving expertise sharing through checkpoint distribution. We note that this approach is currently limited to decoder-only models; preliminary experiments on encoder-based architectures show reduced effectiveness.