Contextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models

TL;DR

This paper tackles multi-task adaptation in large language models by introducing Contextual Attention Modulation (CAM), which dynamically modulates self-attention representations to better fuse general knowledge with task-specific context. It then presents HyCAM, a hybrid framework that combines a shared full-parameter CAM with multiple specialized, parameter-efficient CAMs, guided by a dynamic soft-routing and load-balancing mechanism. Empirical results across five diverse domains and multiple backbone models show HyCAM achieving a 3.65% average improvement over strong baselines, with larger models benefiting most from the approach. The work provides a scalable, efficient pathway for robust multi-task LLM adaptation and offers insights into attention-centered modulation as a principled mechanism for preserving knowledge while enabling specialization.

Abstract

Large Language Models (LLMs) possess remarkable generalization capabilities but struggle with multi-task adaptation, particularly in balancing knowledge retention with task-specific specialization. Conventional fine-tuning methods suffer from catastrophic forgetting and substantial resource consumption, while existing parameter-efficient methods perform suboptimally in complex multi-task scenarios. To address this, we propose Contextual Attention Modulation (CAM), a novel mechanism that dynamically modulates the representations of self-attention modules in LLMs. CAM enhances task-specific features while preserving general knowledge, thereby facilitating more effective and efficient adaptation. For effective multi-task adaptation, CAM is integrated into our Hybrid Contextual Attention Modulation (HyCAM) framework, which combines a shared, full-parameter CAM module with multiple specialized, lightweight CAM modules, enhanced by a dynamic routing strategy for adaptive knowledge fusion. Extensive experiments on heterogeneous tasks, including question answering, code generation, and logical reasoning, demonstrate that our approach significantly outperforms existing approaches, achieving an average performance improvement of 3.65%. The implemented code and data are available to ease reproducibility at https://github.com/Applied-Machine-Learning-Lab/HyCAM.

Figures (5)

• Figure 1: The architecture of the CAM and HyCAM framework. HyCAM applies a hybrid CAM mechanism to the output of the Attention module within each Transformer block, while the backbone LLM remains frozen. Specifically, HyCAM integrates a shared, full-parameter CAM module and multiple lightweight Specialized CAMs for common and task-specific knowledge.
• Figure 2: Scatter plots of attention representations. Higher density indicates improved representation capacity of the attention module.
• Figure 3: Impact of Number of Specialized CAMs.
• Figure 4: The Weight matrix of HyCAM.
• Figure 5: Comparison of the training processes across different methods.