Self-Updatable Large Language Models by Integrating Context into Model Parameters

TL;DR

This work tackles the challenge of integrating in-time, small-scale experiences into large language models without increasing parameter or storage overhead. It introduces SELF-PARAM, which embeds context directly into model parameters by minimizing the $KL$ divergence between a context-enabled original model and a context-free target model, using a context-aware target sentence set generated from an instruct model and regularized with unrelated data. Through extensive experiments on single, batch, sequential context injections and conversational recommendation, SELF-PARAM consistently outperforms strong baselines and approaches the upper bound set by context-plus-question prompts, all with zero additional storage. The results demonstrate a scalable, efficient approach to evolving LLMs in dynamic environments, enabling robust memory of recent events and retention of past experiences for long-term applicability.

Abstract

Despite significant advancements in large language models (LLMs), the rapid and frequent integration of small-scale experiences, such as interactions with surrounding objects, remains a substantial challenge. Two critical factors in assimilating these experiences are (1) Efficacy: the ability to accurately remember recent events; (2) Retention: the capacity to recall long-past experiences. Current methods either embed experiences within model parameters using continual learning, model editing, or knowledge distillation techniques, which often struggle with rapid updates and complex interactions, or rely on external storage to achieve long-term retention, thereby increasing storage requirements. In this paper, we propose SELF-PARAM (Self-Updatable Large Language Models with Parameter Integration). SELF-PARAM requires no extra parameters while ensuring near-optimal efficacy and long-term retention. Our method employs a training objective that minimizes the Kullback-Leibler (KL) divergence between the predictions of an original model (with access to contextual information) and a target model (without such access). By generating diverse question-answer pairs related to the knowledge and minimizing the KL divergence across this dataset, we update the target model to internalize the knowledge seamlessly within its parameters. Evaluations on question-answering and conversational recommendation tasks demonstrate that SELF-PARAM significantly outperforms existing methods, even when accounting for non-zero storage requirements. This advancement paves the way for more efficient and scalable integration of experiences in large language models by embedding knowledge directly into model parameters.

Figures (5)

• Figure 1: The Process of Context Injection. Given the original model $\theta$ and a context $x$, our goal is to obtain a new model $\theta'$ that incorporates $x$ directly into its parameters. To achieve this, we require that for any prompt $p$, the model $\theta'$ generates the same output as the original model $\theta$ would when provided with the combined input $x + p$. In other words, $\theta'$ should emulate $\theta$'s behavior with access to the context $x$ when only prompted by $p$. For a random sentence $s$, let $P(s \mid x + p, \theta)$ denote the token distribution generated by the original model $\theta$ when prompted with $x + p$, and let $P(s \mid p, \theta')$ represent the distribution from the new model $\theta'$ when prompted with $p$. Ideally, for all possible sentences $s$, we aim to minimize the KL divergence between $P(s \mid x + p, \theta)$ and $P(s \mid p, \theta')$. This ensures that $\theta'$ accurately integrates the context $x$ within its parameters.
• Figure 2: Average QA-F1 scores after sequentially injecting contexts into the model across 50 sequences. For each sequence, 20 contexts are injected one by one. The first column (step 0) represents the performance of the base model when queried without any injected context. Each subsequent column (step $i$, where $1 \leq i \leq 20$) shows the model’s QA-F1 score on each of the contexts across all 20 contexts after $i$ injection steps. The displayed scores are the mean values averaged over all 50 sequences, demonstrating the model’s retention ability as contexts are progressively injected.
• Figure 3: Average QA-F1 scores after sequentially injecting contexts into the model across 50 sequences without SlimPajama
• Figure 4: Average QA-F1 scores after sequentially injecting contexts into the model across 50 sequences with fine-tuning on the Target Sentence Set.
• Figure 5: Average QA-F1 scores after sequentially injecting contexts into the model across 50 sequences with fine-tuning on the context.