Self-Adapting Language Models

TL;DR

This work tackles the rigidity of pretrained LLMs by enabling self-directed adaptation through self-generated training data and update directives. It introduces SEAL, a two-loop framework where a model generates self-edits and uses RL to maximize downstream performance after applying those edits, with an inner gradient-based update via supervised finetuning. Across knowledge incorporation and ARC few-shot learning, SEAL achieves notable gains over in-context learning and non-RL baselines, and demonstrates CPT benefits in knowledge tasks. The approach suggests a path toward continual, agentic LLMs that can autonomously augment their knowledge and capabilities with synthetic data, reducing reliance on external supervision.

Abstract

Large language models (LLMs) are powerful but static; they lack mechanisms to adapt their weights in response to new tasks, knowledge, or examples. We introduce Self-Adapting LLMs (SEAL), a framework that enables LLMs to self-adapt by generating their own finetuning data and update directives. Given a new input, the model produces a self-edit-a generation that may restructure the information in different ways, specify optimization hyperparameters, or invoke tools for data augmentation and gradient-based updates. Through supervised finetuning (SFT), these self-edits result in persistent weight updates, enabling lasting adaptation. To train the model to produce effective self-edits, we use a reinforcement learning loop with the downstream performance of the updated model as the reward signal. Unlike prior approaches that rely on separate adaptation modules or auxiliary networks, SEAL directly uses the model's own generation to control its adaptation process. Experiments on knowledge incorporation and few-shot generalization show that SEAL is a promising step toward language models capable of self-directed adaptation. Our website and code is available at https://jyopari.github.io/posts/seal.

Figures (7)

• Figure 1: Overview of SEAL. In each RL outer loop iteration, the model generates candidate self-edits (SE)---directives on how to update the weights---applies updates, evaluates performance on a downstream task, and uses the resulting rewards to improve the self-edit generation policy.
• Figure 2: Knowledge Incorporation Setup. Given a new passage, the model generates synthetic data (the self-edit) in the form of "implications" of the passage. We then finetune on these outputs using LoRA. The updated model is evaluated on questions about the passage without access to the original text, and the resulting accuracy serves as the reward signal for reinforcement learning.
• Figure 3: Few-Shot Learning with SEAL. Left: example ARC demonstrations. Center: the model generates a self-edit specifying augmentations and training hyperparameters. Right: the adapted model is evaluated on a held-out test input.
• Figure 4: Accuracy over RL iterations. Each iteration consists of a minibatch of $50$ contexts, each with $5$ sampled self-edits. SEAL surpasses GPT-4.1 synthetic data after two iterations of ReST$^\text{EM}$ on the no-context SQuAD set.
• Figure 5: Example Knowledge Incorporation Self-Edits Across RL Iterations. In this example, we see how RL leads to the generation of more detailed self-edits, which in turn results in better performance. While the progression is clear in this case, the differences across iterations are sometimes more subtle in other examples. We show in §\ref{['app:prompting']} that prompting for longer self-edits is effective, and that RL training further improves performance by a similar margin.