Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Implicit Federated In-context Learning For Task-Specific LLM Fine-Tuning

Dongcheng Li, Junhan Chen, Aoxiang Zhou, Chunpei Li, Youquan Xian, Peng Liu, Xianxian Li

TL;DR

The paper tackles data scarcity and privacy in adapting large language models by introducing IFed-ICL, a three-stage framework that converts private client data into compact context vectors and jointly calibrates small injection coefficients via federated aggregation. By decoupling data utilization from full model training, IFed-ICL enables a single linear context-injection during inference, drastically reducing client-side computation and communication while delivering strong task-specific performance on text classification across multiple LLMs and datasets. Key contributions include the explicit two-part federation (context-vector aggregation and coefficient calibration), a caching-enabled architecture with conventional and vector databases, and empirical evidence of substantial efficiency gains (1.8 KB per round vs MB-scale transfers) alongside robust accuracy improvements. The approach offers a practical, privacy-preserving pathway for distributed, task-specific adaptation of large language models in resource-constrained environments.

Abstract

As large language models continue to develop and expand, the extensive public data they rely on faces the risk of depletion. Consequently, leveraging private data within organizations to enhance the performance of large models has emerged as a key challenge. The federated learning paradigm, combined with model fine-tuning techniques, effectively reduces the number of trainable parameters. However,the necessity to process high-dimensional feature spaces results in substantial overall computational overhead. To address this issue, we propose the Implicit Federated In-Context Learning (IFed-ICL) framework. IFed-ICL draws inspiration from federated learning to establish a novel distributed collaborative paradigm, by converting client local context examples into implicit vector representations, it enables distributed collaborative computation during the inference phase and injects model residual streams to enhance model performance. Experiments demonstrate that our proposed method achieves outstanding performance across multiple text classification tasks. Compared to traditional methods, IFed-ICL avoids the extensive parameter updates required by conventional fine-tuning methods while reducing data transmission and local computation at the client level in federated learning. This enables efficient distributed context learning using local private-domain data, significantly improving model performance on specific tasks.

Implicit Federated In-context Learning For Task-Specific LLM Fine-Tuning

TL;DR

The paper tackles data scarcity and privacy in adapting large language models by introducing IFed-ICL, a three-stage framework that converts private client data into compact context vectors and jointly calibrates small injection coefficients via federated aggregation. By decoupling data utilization from full model training, IFed-ICL enables a single linear context-injection during inference, drastically reducing client-side computation and communication while delivering strong task-specific performance on text classification across multiple LLMs and datasets. Key contributions include the explicit two-part federation (context-vector aggregation and coefficient calibration), a caching-enabled architecture with conventional and vector databases, and empirical evidence of substantial efficiency gains (1.8 KB per round vs MB-scale transfers) alongside robust accuracy improvements. The approach offers a practical, privacy-preserving pathway for distributed, task-specific adaptation of large language models in resource-constrained environments.

Abstract

As large language models continue to develop and expand, the extensive public data they rely on faces the risk of depletion. Consequently, leveraging private data within organizations to enhance the performance of large models has emerged as a key challenge. The federated learning paradigm, combined with model fine-tuning techniques, effectively reduces the number of trainable parameters. However,the necessity to process high-dimensional feature spaces results in substantial overall computational overhead. To address this issue, we propose the Implicit Federated In-Context Learning (IFed-ICL) framework. IFed-ICL draws inspiration from federated learning to establish a novel distributed collaborative paradigm, by converting client local context examples into implicit vector representations, it enables distributed collaborative computation during the inference phase and injects model residual streams to enhance model performance. Experiments demonstrate that our proposed method achieves outstanding performance across multiple text classification tasks. Compared to traditional methods, IFed-ICL avoids the extensive parameter updates required by conventional fine-tuning methods while reducing data transmission and local computation at the client level in federated learning. This enables efficient distributed context learning using local private-domain data, significantly improving model performance on specific tasks.

Paper Structure

This paper contains 16 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Fine-Tuning
  4. In-Context Learning
  5. Proposed Framework
  6. System Setup
  7. Phase 1: Context Vector Extraction and Upload
  8. Stage 2: Global Context Vector Aggregation and Coefficient Calibration
  9. Stage 3: Global Calibration Coefficients Injection
  10. Experiments
  11. Experimental Setup
  12. Experimental Design
  13. Performance Comparison
  14. Efficiency and Communication Evaluation
  15. Impact of Federated Aggregation on Injection Coefficient Performance
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overall of IFed-ICL framework.First stage: Each client extracts vector representations from its private dataset $\mathcal{D}_k$ and transmits them to the server for aggregation. Second stage: Clients collaboratively calibrate the injection coefficients $\Lambda$ by minimizing the perplexity loss on their local data in coordination with the server. Third stage: The trained injection coefficients and the aggregated global context vector are utilized to enable in-context learning for the large language model.
  • Figure 2: Performance comparison between local and global injection coefficients. The figure shows the accuracy of each client when using its locally optimized coefficient versus the globally aggregated coefficient obtained through federated averaging.
  • Figure 3: Analysis of the impact of injection coefficient optimization on performance. The figure illustrates how model performance evolves as the injection coefficient is optimized over successive communication rounds.