Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demonstration-Free Robotic Control via LLM Agents

Brian Y. Tsui, Alan Y. Fang, Tiffany J. Hwu

TL;DR

This work questions the necessity of task-specific demonstrations for robotic manipulation by testing a general-purpose frontier MLLM agent (FAEA) with privileged state access. By treating manipulation as iterative program synthesis in a ReAct-like loop, FAEA achieves high success without demonstrations, achieving LIBERO 84.9%, ManiSkill3 85.7%, and MetaWorld 96% under privileged state access; coaching further improves LIBERO to 88.2%. Across LIBERO, ManiSkill3, and MetaWorld, results demonstrate that deliberative, task-level planning can be effectively handled by frontier agents, enabling data-efficient trajectory generation and potential use as a data augmentation source for VLA training. The study highlights limitations in precision tasks and real-world deployment, while outlining a path to integrate frontier-model advances with robotic perception and safety guardrails for practical embodied systems.

Abstract

Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift. We investigate whether general-purpose large language model (LLM) agent frameworks, originally developed for software engineering, can serve as an alternative control paradigm for embodied manipulation. We introduce FAEA (Frontier Agent as Embodied Agent), which applies an LLM agent framework directly to embodied manipulation without modification. Using the same iterative reasoning that enables software agents to debug code, FAEA enables embodied agents to reason through manipulation strategies. We evaluate an unmodified frontier agent, Claude Agent SDK, across the LIBERO, ManiSkill3, and MetaWorld benchmarks. With privileged environment state access, FAEA achieves success rates of 84.9%, 85.7%, and 96%, respectively. This level of task success approaches that of VLA models trained with less than 100 demonstrations per task, without requiring demonstrations or fine-tuning. With one round of human feedback as an optional optimization, performance increases to 88.2% on LIBERO. This demonstration-free capability has immediate practical value: FAEA can autonomously explore novel scenarios in simulation and generate successful trajectories for training data augmentation in embodied learning. Our results indicate that general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning. This opens a path for robotics systems to leverage actively maintained agent infrastructure and benefit directly from ongoing advances in frontier models. Code is available at https://github.com/robiemusketeer/faea-sim

Demonstration-Free Robotic Control via LLM Agents

TL;DR

This work questions the necessity of task-specific demonstrations for robotic manipulation by testing a general-purpose frontier MLLM agent (FAEA) with privileged state access. By treating manipulation as iterative program synthesis in a ReAct-like loop, FAEA achieves high success without demonstrations, achieving LIBERO 84.9%, ManiSkill3 85.7%, and MetaWorld 96% under privileged state access; coaching further improves LIBERO to 88.2%. Across LIBERO, ManiSkill3, and MetaWorld, results demonstrate that deliberative, task-level planning can be effectively handled by frontier agents, enabling data-efficient trajectory generation and potential use as a data augmentation source for VLA training. The study highlights limitations in precision tasks and real-world deployment, while outlining a path to integrate frontier-model advances with robotic perception and safety guardrails for practical embodied systems.

Abstract

Robotic manipulation has increasingly adopted vision-language-action (VLA) models, which achieve strong performance but typically require task-specific demonstrations and fine-tuning, and often generalize poorly under domain shift. We investigate whether general-purpose large language model (LLM) agent frameworks, originally developed for software engineering, can serve as an alternative control paradigm for embodied manipulation. We introduce FAEA (Frontier Agent as Embodied Agent), which applies an LLM agent framework directly to embodied manipulation without modification. Using the same iterative reasoning that enables software agents to debug code, FAEA enables embodied agents to reason through manipulation strategies. We evaluate an unmodified frontier agent, Claude Agent SDK, across the LIBERO, ManiSkill3, and MetaWorld benchmarks. With privileged environment state access, FAEA achieves success rates of 84.9%, 85.7%, and 96%, respectively. This level of task success approaches that of VLA models trained with less than 100 demonstrations per task, without requiring demonstrations or fine-tuning. With one round of human feedback as an optional optimization, performance increases to 88.2% on LIBERO. This demonstration-free capability has immediate practical value: FAEA can autonomously explore novel scenarios in simulation and generate successful trajectories for training data augmentation in embodied learning. Our results indicate that general-purpose agents are sufficient for a class of manipulation tasks dominated by deliberative, task-level planning. This opens a path for robotics systems to leverage actively maintained agent infrastructure and benefit directly from ongoing advances in frontier models. Code is available at https://github.com/robiemusketeer/faea-sim
Paper Structure (39 sections, 2 equations, 2 figures, 6 tables)

This paper contains 39 sections, 2 equations, 2 figures, 6 tables.

Table of Contents

  1. Motivation and Related Work
  2. LLM-Robot Integration via Custom Pipelines
  3. From Software Agents to Embodied Agents
  4. FAEA Method Overview
  5. Terminology
  6. Formal Framework
  7. Prompt Template
  8. Trace Validation
  9. Simulation Evaluation
  10. Experimental Setup
  11. Simulation Interface
  12. Observation Modality and Comparison Context
  13. Model Selection
  14. Benchmarks
  15. Exploring FAEA Prompt Strategies on LIBERO
  16. ...and 24 more sections

Figures (2)

  • Figure 1: FAEA prompt template. Black text is used for baseline FAEA prompting; blue text shows coaching tips added for the enhanced variant. File paths and environment-specific details are masked.
  • Figure 2: FAEA Architecture. The Claude Agent SDK orchestrates the ReAct loop: reasoning about the task, writing Python scripts, and observing execution results from LIBERO/ManiSkill simulations via Gymnasium interface.