How Much Knowledge Can You Pack into a LoRA Adapter without Harming LLM?

TL;DR

This work studies how many new facts can be injected into a large language model via Low-Rank Adaptation (LoRA) without eroding pre-existing knowledge. By tuning Llama-3.1-8B-Instruct with varying counts of Unknown, MaybeKnown, and HighlyKnown facts and employing paraphrase augmentation, the authors show that a blend of Unknown and HighlyKnown maximizes knowledge uptake but can degrade external reasoning benchmarks like MMLU while improving TruthfulQA performance with paraphrase. They demonstrate that highly biased or overrepresented training data can cause negative shifts and reduced uncertainty, underscoring the need for careful data composition and tuning hyperparameters. The work provides a detailed taxonomy of knowledge states, a suite of intrinsic and extrinsic evaluations, and cross-model validation on Mistral, offering practical guidance for LoRA-based knowledge updates in production systems.

Abstract

The performance of Large Language Models (LLMs) on many tasks is greatly limited by the knowledge learned during pre-training and stored in the model's parameters. Low-rank adaptation (LoRA) is a popular and efficient training technique for updating or domain-specific adaptation of LLMs. In this study, we investigate how new facts can be incorporated into the LLM using LoRA without compromising the previously learned knowledge. We fine-tuned Llama-3.1-8B-instruct using LoRA with varying amounts of new knowledge. Our experiments have shown that the best results are obtained when the training data contains a mixture of known and new facts. However, this approach is still potentially harmful because the model's performance on external question-answering benchmarks declines after such fine-tuning. When the training data is biased towards certain entities, the model tends to regress to few overrepresented answers. In addition, we found that the model becomes more confident and refuses to provide an answer in only few cases. These findings highlight the potential pitfalls of LoRA-based LLM updates and underscore the importance of training data composition and tuning parameters to balance new knowledge integration and general model capabilities.

Figures (5)

• Figure 1: Decrease in quality with increase of new facts learned by the model: results of the fine-tuned Llama-3.1-8B-Instruct on TruthfulQA (solid line corresponds to the mean score, error margin -- to the min/max scores of three runs with different random seeds).
• Figure 2: Dynamics of the reliability score during training on 500 (left) and 3,000 (right) Unknown items along with paraphrases and HighlyKnown facts. Error bar is min-max for 3 seed run.
• Figure 3: MMLU: Accuracy dependent on the amount of Unknown learned. Pointed horizontal line indicates the baseline. Models trained with less additional data tend to disrupt reasoning less.
• Figure 4: TruthfulQA: MC1 and MC2 accuracy metrics dependent on the amount of Unknown learned. Horizontal dotted lines indicate the baselines. Models trained with paraphrases tend to disrupt truthfulness less. Error bar is min-max for 3 seed run.
• Figure 5: Difference in quality drop for the two models: Llama 3.1 8B and Mistral 7B v0.3.