Self-Regulation and Requesting Interventions

TL;DR

The paper tackles reliable LLM-based agents under a constrained intervention budget by marrying LLM process reward models (PRMs) with classical tabular reinforcement learning in an offline framework. It introduces a three-phase pipeline: (i) offline transition collection and PRM training for self-regulation, (ii) dynamic-programming-based reward and usage search to derive optimal intervention policies under budget $C$, and (iii) supervised fine-tuning of a helper model to imitate the DP-derived policy. The method balances task success with intervention cost, achieving near-parallel performance to always-intervene baselines while using far fewer interventions on Situated Instruction Following tasks. This approach advances trustworthy AI by enabling self-regulation and targeted assistance, with practical impact for deploying reliable LLM agents in uncertain, real-world environments.

Abstract

Human intelligence involves metacognitive abilities like self-regulation, recognizing limitations, and seeking assistance only when needed. While LLM Agents excel in many domains, they often lack this awareness. Overconfident agents risk catastrophic failures, while those that seek help excessively hinder efficiency. A key challenge is enabling agents with a limited intervention budget $C$ is to decide when to request assistance. In this paper, we propose an offline framework that trains a "helper" policy to request interventions, such as more powerful models or test-time compute, by combining LLM-based process reward models (PRMs) with tabular reinforcement learning. Using state transitions collected offline, we score optimal intervention timing with PRMs and train the helper model on these labeled trajectories. This offline approach significantly reduces costly intervention calls during training. Furthermore, the integration of PRMs with tabular RL enhances robustness to off-policy data while avoiding the inefficiencies of deep RL. We empirically find that our method delivers optimal helper behavior.

Figures (6)

• Figure 1: Unreliable agents and training challenges.(a) An unreliable agent neither completes the assigned task nor communicates its inability, causing surprise and catastrophe. (b) When there is a budget $C$ on interventions requested during inference, a key challenge is determining a reward function that guides the agent to request help appropriately. (c) For both behavior cloning/reinforcement learning, obtaining an optimal demonstration is complicated by the exponential space of possible trajectories, difficult even with human effort.
• Figure 2: (a) A SIF task requires the agent to locate objects, interact with humans, and perform household tasks in a sequence of discrete actions. Assuming perfect visual perception, the relevant segment is highlighted in orange; states are represented in text. (b) A brief overview of Self-Regulation and Requesting Intervention, in comparison to the base agent.
• Figure 3: $p(s)$ measured by the PRM across the task. Interventions on PRM-chosen states (red line and stars) cause repeated toggling that traps the agent in low-$p(s)$ regions, resulting in worse outcomes than random interventions (blue line and stars), which ends at step 10 with task success.
• Figure 4: Method Overview.(a) We combine tabular state dynamics with a process reward model (PRM), implemented as a large language model, to perform offline tabular reinforcement learning. Our method consists of iterative usage/policy computation. $\pi_s=\text{help}$ denoted as $\pi_s=1$ for space. (b) We run this offline tabular RL procedure to generate trajectory annotations for training tasks. (c) Finally, we train another large language model with a scalar head via supervised fine-tuning (SFT), with the trajectory annotations from step (b).
• Figure 5: Example Trajectory with interventions and base actor.