Sensory-Motor Control with Large Language Models via Iterative Policy Refinement

TL;DR

The paper addresses enabling large language models (LLMs) to control embodied agents via direct continuous actuation, bypassing predefined motor primitives. It introduces a two-phase prompting workflow that first elicits a high-level control strategy, converts it into executable rules and Python code, and then iteratively refines the policy using environment feedback and sensory-motor data. Across Gymnasium tasks and MuJoCo Pendulum, the method yields optimal or near-optimal policies using open-weight models (GPT-oss:120b, Qwen2.5:72b); ablations show the importance of including sensory-motor data and structured prompts. The results demonstrate that open LLMs can serve as effective priors and reasoning engines for embodied control without fine-tuning, offering a path toward data-efficient, plug-and-play robot control and suggesting future directions like adaptive prompting and dual-model systems.

Abstract

We propose a method that enables large language models (LLMs) to control embodied agents through the generation of control policies that directly map continuous observation vectors to continuous action vectors. At the outset, the LLMs generate a control strategy based on a textual description of the agent, its environment, and the intended goal. This strategy is then iteratively refined through a learning process in which the LLMs are repeatedly prompted to improve the current strategy, using performance feedback and sensory-motor data collected during its evaluation. The method is validated on classic control tasks from the Gymnasium library and the inverted pendulum task from the MuJoCo library. The approach proves effective with relatively compact models such as GPT-oss:120b and Qwen2.5:72b. In most cases, it successfully identifies optimal or near-optimal solutions by integrating symbolic knowledge derived through reasoning with sub-symbolic sensory-motor data gathered as the agent interacts with its environment.

Figures (8)

• Figure 1: Overview of the prompting method enabling LLMs to directly control the actuators of embodied agents.
• Figure 2: Average reward obtained for the CartPole1 (left) and CartPole2 (right) tasks. The horizontal axis represents the temperature value, while the vertical axis shows the average reward, computed over 10 replications. Line colors indicate the different models evaluated.
• Figure 3: Average reward at the optimal temperature setting for the Cartpole*1 and Cartpole*2 tasks (left and right, respectively). The vertical axis represents the average reward, while the horizontal axis specifies the LLM used.
• Figure 4: Performance metrics for the CartPole *1 (top) and CartPole *2 (bottom) tasks. The plots on the left display the Figure of Merit, while the plots on the right show Success, Robustness, and Learning Time. Histogram colors indicate the LLM used. Results are reported for the optimal temperature setting. Note that the y-axis scales differ between the left-hand plots.
• Figure 5: Experiments conducted by varying the amount of sensory-motor data provided to the model. All experiments were performed on the Pendulum task. The green and orange boxplots correspond to the gpt-oss:20b and gpt-oss:120b models, respectively. Each boxplot summarizes the results of 10 replications.