Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation

Vaishak Kumar

TL;DR

Act-Observe-Rewrite (AOR), a framework in which an LLM agent improves a robot manipulation policy by synthesising entirely new executable Python controller code between trials, guided by visual observations and structured episode outcomes, is presented.

Abstract

Can a multimodal language model learn to manipulate physical objects by reasoning about its own failures-without gradient updates, demonstrations, or reward engineering? We argue the answer is yes, under conditions we characterise precisely. We present Act-Observe-Rewrite (AOR), a framework in which an LLM agent improves a robot manipulation policy by synthesising entirely new executable Python controller code between trials, guided by visual observations and structured episode outcomes. Unlike prior work that grounds LLMs in pre-defined skill libraries or uses code generation for one-shot plan synthesis, AOR makes the full low-level motor control implementation the unit of LLM reasoning, enabling the agent to change not just what the robot does, but how it does it. The central claim is that interpretable code as the policy representation creates a qualitatively different kind of in-context learning from opaque neural policies: the agent can diagnose systematic failures and rewrite their causes. We validate this across three robosuite manipulation tasks and report promising results, with the agent achieving high success rates without demonstrations, reward engineering, or gradient updates.

Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation

TL;DR

Act-Observe-Rewrite (AOR), a framework in which an LLM agent improves a robot manipulation policy by synthesising entirely new executable Python controller code between trials, guided by visual observations and structured episode outcomes, is presented.

Abstract

Can a multimodal language model learn to manipulate physical objects by reasoning about its own failures-without gradient updates, demonstrations, or reward engineering? We argue the answer is yes, under conditions we characterise precisely. We present Act-Observe-Rewrite (AOR), a framework in which an LLM agent improves a robot manipulation policy by synthesising entirely new executable Python controller code between trials, guided by visual observations and structured episode outcomes. Unlike prior work that grounds LLMs in pre-defined skill libraries or uses code generation for one-shot plan synthesis, AOR makes the full low-level motor control implementation the unit of LLM reasoning, enabling the agent to change not just what the robot does, but how it does it. The central claim is that interpretable code as the policy representation creates a qualitatively different kind of in-context learning from opaque neural policies: the agent can diagnose systematic failures and rewrite their causes. We validate this across three robosuite manipulation tasks and report promising results, with the agent achieving high success rates without demonstrations, reward engineering, or gradient updates.
Paper Structure (39 sections, 1 equation, 1 figure, 3 tables)

This paper contains 39 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Contribution:
  3. Background and Problem Formulation
  4. The In-Context Policy Learning Problem
  5. Related Work
  6. Verbal Self-Reflection for Agents
  7. Code Generation for Robot Control
  8. LLM-Generated Reward Functions and Fitness Criteria
  9. LLM-Based Optimisation
  10. Grounded Planning with LLMs
  11. Foundation Models and VLAs for Manipulation
  12. In-Context Learning and Adaptation in Robotics
  13. The AOR Framework
  14. The Four Components
  15. Vision pipeline:
  16. ...and 24 more sections

Figures (1)

  • Figure 1: The AOR two-timescale loop. The Policy (green box)---comprising the vision pipeline, feature extraction, and Python controller---is the unit synthesised by the LLM after each episode. Fast loop (top): sensor observations flow through the Policy to produce actions each timestep, repeating at control frequency throughout an episode. Slow loop (bottom): at episode end, outcomes and key-frame images enter episodic memory; the multimodal LLM diagnoses failure modes and synthesises an entirely new Policy, dynamically compiled for the next trial. No model weights are updated at any point.