Enhancing Robustness in Language-Driven Robotics: A Modular Approach to Failure Reduction

TL;DR

This work addresses robustness gaps in language-driven robotics, particularly when using smaller language models that are suitable for local deployment. It formalizes planning as a goal-conditioned POMDP, identifies two failure modes (subgoal mischaracterization and missing subgoals), and introduces an explicit expected-outcomes module plus a real-time feedback loop to recover from errors. The proposed modular architecture, implemented in ROS, demonstrates improved task success on pick-and-place and manipulation tasks in both simulation and hardware, outperforming larger LLM baselines while reducing computational costs. The results show that smaller, locally executable LLMs can support robust, scalable robotic task execution with explainable planning and efficient runtime performance.

Abstract

Recent advances in large language models (LLMs) have led to significant progress in robotics, enabling embodied agents to better understand and execute open-ended tasks. However, existing approaches using LLMs face limitations in grounding their outputs within the physical environment and aligning with the capabilities of the robot. This challenge becomes even more pronounced with smaller language models, which are more computationally efficient but less robust in task planning and execution. In this paper, we present a novel modular architecture designed to enhance the robustness of LLM-driven robotics by addressing these grounding and alignment issues. We formalize the task planning problem within a goal-conditioned POMDP framework, identify key failure modes in LLM-driven planning, and propose targeted design principles to mitigate these issues. Our architecture introduces an ``expected outcomes'' module to prevent mischaracterization of subgoals and a feedback mechanism to enable real-time error recovery. Experimental results, both in simulation and on physical robots, demonstrate that our approach significantly improves task success rates for pick-and-place and manipulation tasks compared to both larger LLMs and standard baselines. Through hardware experiments, we also demonstrate how our architecture can be run efficiently and locally. This work highlights the potential of smaller, locally-executable LLMs in robotics and provides a scalable, efficient solution for robust task execution.

Figures (5)

• Figure 1: Overview of the proposed approach: When receiving instructions from a user, a plan and the expected outcomes (EOs) of each step are inferred. The robot then tackles the task iteratively, using motion primitives and receiving feedback from a simulation, before execution in the environment. See Section \ref{['sec:formalism']} for notations and formalism.
• Figure 2: Options $\pi_1$ and $\pi_2$ (red) being parameterized versions of option $\pi$ (blue). The outcome sets of $\pi_1$ and $\pi_2$ are subsets of the outcome set of $\pi$. For example, $\pi$ might be "grasp mug", with $\pi_1$ being "grasp mug handle" and $\pi_2$ "grasp mug body". $O$ denotes the outcome set of $\pi$, while $O_1$ and $O_2$, respectively, denote the outcome sets of $\pi_1$ and $\pi_2$.
• Figure 3: Common errors for LLM plan execution. The GTSGs are plain blue ellipses while the plan subgoals are represented by dark blue contours. The options are in red (we omit the initialization and outcome sets of the options for legibility). Left: the plan subgoals and GTSGs are aligned. Center: Mischaracterized subgoal: while the state achieves the plan subgoal, it does not fulfil the GTSG and does not afford the next primitive. Right: Missing subgoal in the plan: the plan step requires more than one option to be carried out, leading to execution errors.
• Figure 4: Successful execution of the "Move the Water Glass to the Coffee table. It is currently on the Kitchen table." task. From left to right, top to bottom: (i) the service robot approaches the kitchen table; (ii) the robot grasps the glass of water; (iii) the robot moves to the coffee table, (iv) the task ends with the robot putting down the glass.
• Figure 5: Successful execution of the "Plug in the charger" task. From left to right, top to bottom: (i) the gripper approaches the charger's plug; (ii) the charger is grasped by the gripper; (iii) the arm begins plugging the charger, (iv) the task ends with the charger plugged.