CLLoRA: An Approach to Measure the Effects of the Context Length for LLM Fine-Tuning

TL;DR

This work tackles the challenge of quantifying how context length and data quantity influence fine-tuning of large language models in privacy-preserving federated settings with non-IID data. It introduces CLLoRA, a framework that combines LoRA-based parameter-efficient fine-tuning with Dirichlet-based non-IID data partitioning across two axes: context length and number of sentences. Key findings show that context quantity imbalance more strongly affects the global model, while context length exerts a larger effect on global performance with small models being more sensitive to length and large models showing robustness; LoRA also dramatically reduces parameter count and communication. The approach provides principled benchmarks for non-IID textual data in federated fine-tuning and demonstrates practical gains in efficiency for privacy-preserving adaptation of LLMs.

Abstract

Large language model fine-tuning has been identified as an efficient approach to applying the pre-trained Large language models to other domains. To guarantee data privacy for different data owners, models are often fine-tuned in federated learning environments across different data owners, which often involve data heterogeneity issues and affect the fine-tuning performance. In addition, the length of the context for the training data has been identified as a major factor that affects the LLM's model performance. To efficiently measure how the context length affects the LLM's model performance in heterogeneous federated learning environments, we propose CLLoRA. CLLoRA utilizes the parameter-efficient fine-tuning approach LoRA based on different kinds of LLMs with varying sizes as the fine-tuning approach to investigate whether the quality and length of contexts can serve as standards for measuring non-IID context. The findings indicate that an imbalance in context quality not only affects local training on clients but also impacts the global model's performance. However, context length has a minimal effect on local training but a more significant influence on the global model. These results provide insights into how context quality and length affect the model performance for LLM fine-tuning in federated learning environments.

Figures (9)

• Figure 1: The process of horizontal federated learning.
• Figure 2: The design strategy of LoRA.
• Figure 3: 3D Scatter Plot of Dirichlet Distribution.
• Figure 4: Different $\alpha$ for label distribution skew.
• Figure 5: Different $\alpha$ for quantity distribution skew.