ABBEL: LLM Agents Acting through Belief Bottlenecks Expressed in Language

TL;DR

AB BEL addresses the challenge of maintaining context in long-horizon tasks by compressing histories into language-based belief states and acting from those beliefs. The authors propose a two-stage ABBEL framework and demonstrate that LLMs can generate interpretable, compact beliefs that keep memory usage nearly constant across steps, with six diverse environments used for evaluation. They show that in some cases belief bottlenecks propagate errors and can underperform full-context baselines, motivating reinforcement learning to tune belief generation and usage. The RL extensions include belief length penalties and belief grading, which, in several tasks, enable ABBEL to outperform full-context baselines while reducing memory demands, highlighting a path toward more scalable, controllable AI agents.

Abstract

As the length of sequential decision-making tasks increases, it becomes computationally impractical to keep full interaction histories in context. We introduce a general framework for LLM agents to maintain concise contexts through multi-step interaction: Acting through Belief Bottlenecks Expressed in Language (ABBEL), and methods to further improve ABBEL agents with RL post-training. ABBEL replaces long multi-step interaction history by a belief state, i.e., a natural language summary of what has been discovered about task-relevant unknowns. Under ABBEL, at each step the agent first updates a prior belief with the most recent observation from the environment to form a posterior belief, then uses only the posterior to select an action. We systematically evaluate frontier models under ABBEL across six diverse multi-step environments, finding that ABBEL supports generating interpretable beliefs while maintaining near-constant memory use over interaction steps. However, bottleneck approaches are generally prone to error propagation, which we observe causing inferior performance when compared to the full context setting due to errors in belief updating. Therefore, we train LLMs to generate and act on beliefs within the ABBEL framework via reinforcement learning (RL). We experiment with belief grading, to reward higher quality beliefs, as well as belief length penalties to reward more compressed beliefs. Our experiments demonstrate the ability of RL to improve ABBEL's performance beyond the full context setting, while using less memory than contemporaneous approaches.

Figures (10)

• Figure 1: (a) Overview of the belief update and action selection contexts over 3 timesteps under ABBEL, in contrast to the typical multi-step paradigm (Vanilla) or simply prompting for belief generation (Belief Prompting) which keep all past steps in context. (b) an example step of ABBEL in Wordle; actions are word guesses, and observations provide feedback on each letter.
• Figure 2: Behavior of frontier models across environments and frameworks.
• Figure 3: Overview of belief grading. Beliefs, actions and observations generated at timestep $t$ are denoted by $b_t$, $a_t$, and $o_t$, respectively. After collecting trajectories from the current ABBEL policy (a), each step is copied into a new group consisting of the original belief update and a newly generated posterior belief from the same context (b), which are each assigned rewards by a belief grader (c). The grader shown here was customized for Combination Lock, a 3-digit version of Wordle. The policy is finally updated with GRPO using both the trajectory groups and the belief groups.
• Figure 4: Test behavior of Qwen2.5-7B-Instruct trained in Combination Lock. (a) Task success rates over training steps show ABBEL quickly reduces its performance gap with other frameworks, while Belief Grading allows ABBEL to surpass the other frameworks. (b) The beliefs generated by ABBEL Belief Grading initially increase in length but eventually become even shorter over training, and remain significantly shorter than the interaction history beyond the first 2 environment steps.
• Figure 5: Model comparison in multi-objective QA. ABBEL performs closest to the full-context Vanilla model for 4+ objectives, and training with a length penalty on the belief state (ABBEL LP) remains competitive while using much less memory.