From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems

TL;DR

This work develops a rigorous theory for LLM-driven autonomous agents by embedding the Planner-Actor-Reporter (PAR) architecture into a hierarchical RL framework. The LLM Planner operates in a high-level POMDP to generate language-based subgoals, while a pretrained Actor executes low-level actions and a Reporter translates world states into language feedback. It shows that, under suitable pretraining assumptions, the LLM Planner performs Bayesian Aggregated Imitation Learning (BAIL) via in-context learning, but naive reliance on BAIL yields linear regret, which can be mitigated by an ε-greedy exploration strategy achieving sublinear regret. The analysis extends to practical settings where pretraining errors contribute additive terms to the regret, and it further generalizes to scenarios where the LLM acts as a world model (BAWM) and to multi-agent coordination. Overall, the paper offers a principled explanation of how prompting-based planning interacts with real-world exploration and pretraining noise, with implications for designing robust, autonomous LLM-enabled systems.

Abstract

In this work, from a theoretical lens, we aim to understand why large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. To this end, consider a hierarchical reinforcement learning (RL) model where the LLM Planner and the Actor perform high-level task planning and low-level execution, respectively. Under this model, the LLM Planner navigates a partially observable Markov decision process (POMDP) by iteratively generating language-based subgoals via prompting. Under proper assumptions on the pretraining data, we prove that the pretrained LLM Planner effectively performs Bayesian aggregated imitation learning (BAIL) through in-context learning. Additionally, we highlight the necessity for exploration beyond the subgoals derived from BAIL by proving that naively executing the subgoals returned by LLM leads to a linear regret. As a remedy, we introduce an $ε$-greedy exploration strategy to BAIL, which is proven to incur sublinear regret when the pretraining error is small. Finally, we extend our theoretical framework to include scenarios where the LLM Planner serves as a world model for inferring the transition model of the environment and to multi-agent settings, enabling coordination among multiple Actors.

Figures (2)

• Figure 1: Overview of the Planner-Actor-Reporter (PAR) system as LLM Agents. Acting as a central controller, the Planner conducts the high-level planning by storing the history and reasoning through the iterative use of the ICL ability of LLMs, coupled with explorations. The Actor handles low-level planning and executes subgoals using pre-programmed skill sets, and the Reporter perceives and processes multimodal information from environment to reinforce the ongoing planning.
• Figure 2: Illustration of structure of hmdp. The low-level MDP is featured by transition kernel $\mathbb{T}$, which characterizes the dynamics of the physical environment. The high-level transition is a result of a sequence of low-level actions in the physical environment, guided by policies $\mu=\{\mu_g\}_{g\in\mathcal{G}}$. Thus, high-level pomdp incorporates latent information $z=(\mathbb{T},\mu)$ originated from the low-level.

Theorems & Definitions (23)

• Remark 3.1
• Remark 3.2
• Definition 4.1: Perfect Setting
• Proposition 4.2: LLM Performs BAIL
• Proposition 4.3: Hard-to-Distinguish Example
• Definition 4.4: $\eta$-distinguishability
• Theorem 4.6: Regret under Perfect Setting
• Remark 4.7
• Theorem 5.3: zhang2023and
• Theorem 5.5: Pretrained Translator