SAS-Prompt: Large Language Models as Numerical Optimizers for Robot Self-Improvement

TL;DR

The paper tackles whether large language models can drive self-improvement of robot policies without external optimization loops by introducing the SAS Prompt, which retrieves, analyzes, and synthesizes from prior traces to iteratively adjust attenuation parameters $\boldsymbol{\theta} \in \mathbb{R}^8$ that scale the lower-level controller outputs $\mathbf{x} \in \mathbb{R}^8$. It demonstrates gradient-free numerical optimization inside an LLM and validates the approach on a real table-tennis robot and in Mujoco simulation, with retrieval and self-improvement components guiding policy search. Key findings show competitive optimization performance against baselines, successful real-robot and sim-based self-improvement toward task targets, and transparent, justified parameter updates. The work introduces a new family of explainable, fully LLM-based policy search methods with potential to scale toward more complex robotic tasks and environments.

Abstract

We demonstrate the ability of large language models (LLMs) to perform iterative self-improvement of robot policies. An important insight of this paper is that LLMs have a built-in ability to perform (stochastic) numerical optimization and that this property can be leveraged for explainable robot policy search. Based on this insight, we introduce the SAS Prompt (Summarize, Analyze, Synthesize) -- a single prompt that enables iterative learning and adaptation of robot behavior by combining the LLM's ability to retrieve, reason and optimize over previous robot traces in order to synthesize new, unseen behavior. Our approach can be regarded as an early example of a new family of explainable policy search methods that are entirely implemented within an LLM. We evaluate our approach both in simulation and on a real-robot table tennis task. Project website: sites.google.com/asu.edu/sas-llm/

Figures (8)

• Figure 1: An example execution trace in the robot table tennis domain. The positions of the paddle and ball at every time step are provided along with robot control parameters (a -- h).
• Figure 2: Numerical Optimization inside an LLM: the top figures depict two runs of an optimization process leveraging an LLM. The bottom text shows how the LLM is iteratively queried to minimize function $f(x)$. No gradients are provided.
• Figure 3: SAS Prompt: the prompt provides the LLM with information about the domain, the user objectives and a step-by-step instruction on how to summarize and analyze the in-context examples. In turn, the final step is to synthesize a new set of parameters. Step 1 and 2 aim at retrieving previous examples that are best aligned with human objective. Step 3 and 4 aim at further optimizing these values to improve robot performance with regards to the same objective.
• Figure 4: Retrieval results: the retrieval part of the SAS prompt is used to retrieve robot control parameters which best fit the human instructions (printed above). Depicted coordinates are ball landing locations in a real-robot experiment.
• Figure 5: Visualization of the ball trajectory in a retrieval task.