AgenticLab: A Real-World Robot Agent Platform that Can See, Think, and Act

TL;DR

AgenticLab tackles the challenge of evaluating open-world, long-horizon robot manipulation under real-world conditions by introducing a model-agnostic, closed-loop robot agent platform and a real-world benchmark. The system interleaves perception, grounding, planning, execution, verification, and replanning, enabling fair cross-model evaluation and exposing failure modes absent in offline or simulated tests. Key contributions include (i) a reproducible hardware-software platform, (ii) a modular, closed-loop agent framework with verifiable reasoning and multi-view perception, and (iii) a real-world benchmark that reveals grounding and verification limitations across diverse tasks and environments. The findings show that end-to-end success is constrained by the weakest module, dense verification improves robustness at latency cost, and a compositional approach with strong backbones offers practical deployment advantages, with Gemini-based models providing reliable performance on embodied tasks.

Abstract

Recent advances in large vision-language models (VLMs) have demonstrated generalizable open-vocabulary perception and reasoning, yet their real-robot manipulation capability remains unclear for long-horizon, closed-loop execution in unstructured, in-the-wild environments. Prior VLM-based manipulation pipelines are difficult to compare across different research groups' setups, and many evaluations rely on simulation, privileged state, or specially designed setups. We present AgenticLab, a model-agnostic robot agent platform and benchmark for open-world manipulation. AgenticLab provides a closed-loop agent pipeline for perception, task decomposition, online verification, and replanning. Using AgenticLab, we benchmark state-of-the-art VLM-based agents on real-robot tasks in unstructured environments. Our benchmark reveals several failure modes that offline vision-language tests (e.g., VQA and static image understanding) fail to capture, including breakdowns in multi-step grounding consistency, object grounding under occlusion and scene changes, and insufficient spatial reasoning for reliable manipulation. We will release the full hardware and software stack to support reproducible evaluation and accelerate research on general-purpose robot agents.

Figures (8)

• Figure 1: AgenticLab is a model-agnostic real-world robot agent platform that uses onboard cameras and open-vocabulary language prompts to manipulate in unstructured environments, with closed-loop reasoning that interleaves execution, verification, and replanning throughout manipulation.
• Figure 2: The AgenticLab Manipulation Platform: a fully reproducible, easy-to-deploy platform for real-world, in-the-wild robot agent benchmark.
• Figure 3: Pipeline of AgenticLab. AgenticLab executes a closed-loop agentic reasoning pipeline for manipulation that integrates task parsing, grounding, planning, execution, verification, and replanning. As illustrated in the figure, a human provides a natural-language instruction, and the robot first acquires onboard observations and parses the task to produce an initial plan. The agent then follows the green arrows to execute each block in sequence, carrying out each step of the initial plan along this path until all planned steps are completed. Each block includes a VLM-based verification step: if verification succeeds, execution proceeds to the next block along the green path; otherwise, the agent follows the red arrows to retry or replan based on updated observations. For grasp planning, the agent first plans grasps using the shoulder camera; if grasp verification fails, it replans with the wrist camera for more localized visual cues.
• Figure 4: Manipulation tasks used for real-world evaluation and benchmark. These tasks are selected to cover a range of challenges, including spatial reasoning, vision-language grounding, and long-horizon planning.
• Figure 5: Failure mode breakdown for single-VLM pipelines on the sorting task. Results show which pipeline components dominate failures across different VLMs.