Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALRM: Agentic LLM for Robotic Manipulation

Vitor Gaboardi dos Santos, Ibrahim Khadraoui, Ibrahim Farhat, Hamza Yous, Samy Teffahi, Hakim Hacid

TL;DR

ALRM advances robotic manipulation by integrating LLM-driven planning with agentic execution in a ReAct-style loop, offering two execution modalities: Code-as-Policy (CaP) and Tool-as-Policy (TaP). A 56-task Gazebo-based benchmark emphasizes high-level, multi-step reasoning and linguistic variation to systematically evaluate both open- and closed-source LLMs. Across ten LLMs, large models generally benefit from TaP at the cost of latency, while small models shine in CaP, highlighting a trade-off between interpretability, speed, and robustness. The work demonstrates practical potential for language-conditioned robotics and outlines directions for broader evaluation and eventual real-robot deployment.

Abstract

Large Language Models (LLMs) have recently empowered agentic frameworks to exhibit advanced reasoning and planning capabilities. However, their integration in robotic control pipelines remains limited in two aspects: (1) prior \ac{llm}-based approaches often lack modular, agentic execution mechanisms, limiting their ability to plan, reflect on outcomes, and revise actions in a closed-loop manner; and (2) existing benchmarks for manipulation tasks focus on low-level control and do not systematically evaluate multistep reasoning and linguistic variation. In this paper, we propose Agentic LLM for Robot Manipulation (ALRM), an LLM-driven agentic framework for robotic manipulation. ALRM integrates policy generation with agentic execution through a ReAct-style reasoning loop, supporting two complementary modes: Code-asPolicy (CaP) for direct executable control code generation, and Tool-as-Policy (TaP) for iterative planning and tool-based action execution. To enable systematic evaluation, we also introduce a novel simulation benchmark comprising 56 tasks across multiple environments, capturing linguistically diverse instructions. Experiments with ten LLMs demonstrate that ALRM provides a scalable, interpretable, and modular approach for bridging natural language reasoning with reliable robotic execution. Results reveal Claude-4.1-Opus as the top closed-source model and Falcon-H1-7B as the top open-source model under CaP.

ALRM: Agentic LLM for Robotic Manipulation

TL;DR

ALRM advances robotic manipulation by integrating LLM-driven planning with agentic execution in a ReAct-style loop, offering two execution modalities: Code-as-Policy (CaP) and Tool-as-Policy (TaP). A 56-task Gazebo-based benchmark emphasizes high-level, multi-step reasoning and linguistic variation to systematically evaluate both open- and closed-source LLMs. Across ten LLMs, large models generally benefit from TaP at the cost of latency, while small models shine in CaP, highlighting a trade-off between interpretability, speed, and robustness. The work demonstrates practical potential for language-conditioned robotics and outlines directions for broader evaluation and eventual real-robot deployment.

Abstract

Large Language Models (LLMs) have recently empowered agentic frameworks to exhibit advanced reasoning and planning capabilities. However, their integration in robotic control pipelines remains limited in two aspects: (1) prior \ac{llm}-based approaches often lack modular, agentic execution mechanisms, limiting their ability to plan, reflect on outcomes, and revise actions in a closed-loop manner; and (2) existing benchmarks for manipulation tasks focus on low-level control and do not systematically evaluate multistep reasoning and linguistic variation. In this paper, we propose Agentic LLM for Robot Manipulation (ALRM), an LLM-driven agentic framework for robotic manipulation. ALRM integrates policy generation with agentic execution through a ReAct-style reasoning loop, supporting two complementary modes: Code-asPolicy (CaP) for direct executable control code generation, and Tool-as-Policy (TaP) for iterative planning and tool-based action execution. To enable systematic evaluation, we also introduce a novel simulation benchmark comprising 56 tasks across multiple environments, capturing linguistically diverse instructions. Experiments with ten LLMs demonstrate that ALRM provides a scalable, interpretable, and modular approach for bridging natural language reasoning with reliable robotic execution. Results reveal Claude-4.1-Opus as the top closed-source model and Falcon-H1-7B as the top open-source model under CaP.
Paper Structure (21 sections, 2 figures, 3 tables)

This paper contains 21 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-Based Robotic Agent Architecture for Task Planning and Execution
  4. Task Planner Agent
  5. Task Executor Agent
  6. Tool-as-Policy (TaP)
  7. Code-as-Policy (CaP)
  8. API Server for Robotic Control
  9. Benchmark
  10. Environments
  11. Tasks
  12. Paraphrase Categories
  13. Experimental Setup
  14. Large Language Models
  15. Evaluation Metrics
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Proposed LLM-based agent architecture for solving high-level robotic arm manipulation tasks. The architecture consists of three main modules: (1) Task Planner Agent, (2) Task Executor Agent, and (3) API Server.
  • Figure 2: Illustration of the three environments designed for evaluation.