LLM Reinforcement in Context

TL;DR

This paper frames LLM alignment under long-context usage as a scaling problem, driven by the exponential growth of required reinforcement data with context length and the waning influence of fixed system prompts. It proposes interruptions—in-context control statements inserted into the user input and Chain-of-Thought outputs every $t$ tokens—as a weight-free, deployment-level form of reinforcement in context, yielding a nonzero relative influence $\frac{s_i}{t}$ on the total context as $l$ grows. The authors formalize key relationships, including $a_t(l) = \Omega(k^l)$ and $\frac{s}{l} = \frac{s_p}{l} + \frac{s_i}{t}$ with $\lim_{l\to\infty} \frac{s}{l} = \frac{s_i}{t}$, to argue that interruptions can mitigate the scaling problem without retraining. They discuss consequences, trade-offs, and limitations—such as potential performance degradation and the need for operator-controlled deployment—and outline directions for future research and evaluation on alignment benchmarks like harmbench. Overall, the work offers a deployment-focused approach to stabilize LLM alignment in long-context scenarios, with potential applicability to safety-critical applications, while noting that foundational training approaches and user experience considerations require careful management.

Abstract

Current Large Language Model alignment research mostly focuses on improving model robustness against adversarial attacks and misbehavior by training on examples and prompting. Research has shown that LLM jailbreak probability increases with the size of the user input or conversation length. There is a lack of appropriate research into means of strengthening alignment which also scale with user input length. We propose interruptions as a possible solution to this problem. Interruptions are control sentences added to the user input approximately every x tokens for some arbitrary x. We suggest that this can be generalized to the Chain-of-Thought process to prevent scheming.