Context Parametrization with Compositional Adapters
Josip Jukić, Martin Tutek, Jan Šnajder
TL;DR
CompAs introduces a principled framework for context-parametrized adaptation of LLMs by translating multiple contextual sources into composable adapters. Through a teacher–student meta-learning setup, a generator produces additive adapters from individual contexts, enabling algebraic combination to mirror concatenated inputs while reducing inference cost and mitigating long-context instability. The framework is underpinned by a compositionality bound that relates generator additivity and student–teacher alignment to output fidelity, and it uses a reconstruction objective to ensure context recoverability for safety. Empirically, CompAs outperforms ICL and prior generator-based methods on multiple-choice and extractive QA tasks, especially as the number of context sources grows, and demonstrates robustness to noisy or irrelevant context while offering substantial efficiency gains. Overall, compositional adapters provide a scalable, efficient pathway for context-aware LLM deployment with provable and empirical advantages over traditional prompt-based or fine-tuning approaches.
Abstract
Large language models (LLMs) often seamlessly adapt to new tasks through in-context learning (ICL) or supervised fine-tuning (SFT). However, both of these approaches face key limitations: ICL is inefficient when handling many demonstrations, and SFT incurs training overhead while sacrificing flexibility. Mapping instructions or demonstrations from context directly into adapter parameters offers an appealing alternative. While prior work explored generating adapters based on a single input context, it has overlooked the need to integrate multiple chunks of information. To address this gap, we introduce CompAs, a meta-learning framework that translates context into adapter parameters with a compositional structure. Adapters generated this way can be merged algebraically, enabling instructions, demonstrations, or retrieved passages to be seamlessly combined without reprocessing long prompts. Critically, this approach yields three benefits: lower inference cost, robustness to long-context instability, and establishes a principled solution when input exceeds the model's context window. Furthermore, CompAs encodes information into adapter parameters in a reversible manner, enabling recovery of input context through a decoder, facilitating safety and security. Empirical results on diverse multiple-choice and extractive question answering tasks show that CompAs outperforms ICL and prior generator-based methods, especially when scaling to more inputs. Our work establishes composable adapter generation as a practical and efficient alternative for scaling LLM deployment.