Interpretable Robot Control via Structured Behavior Trees and Large Language Models

TL;DR

The paper tackles the challenge of intuitive human–robot interaction in dynamic, real-world environments by uniting Large Language Models with Behavior Trees to translate natural language instructions into executable robot actions via modular plugins. It introduces an open-source, robot-agnostic framework that extends ROSA with autonomous behavior selection, multimodal HRI, failure reasoning, and structured BT-based control, enabling end-to-end command-to-execution flows. The authors provide a detailed evaluation across drone and legged platforms, reporting high end-to-end success rates and robust performance in perception-driven tasks and motion commands. The work advances interpretability and scalability in HRI, showing practical impact for flexible, naturalistic robot operation in unstructured settings.

Abstract

As intelligent robots become more integrated into human environments, there is a growing need for intuitive and reliable Human-Robot Interaction (HRI) interfaces that are adaptable and more natural to interact with. Traditional robot control methods often require users to adapt to interfaces or memorize predefined commands, limiting usability in dynamic, unstructured environments. This paper presents a novel framework that bridges natural language understanding and robotic execution by combining Large Language Models (LLMs) with Behavior Trees. This integration enables robots to interpret natural language instructions given by users and translate them into executable actions by activating domain-specific plugins. The system supports scalable and modular integration, with a primary focus on perception-based functionalities, such as person tracking and hand gesture recognition. To evaluate the system, a series of real-world experiments was conducted across diverse environments. Experimental results demonstrate that the proposed approach is practical in real-world scenarios, with an average cognition-to-execution accuracy of approximately 94%, making a significant contribution to HRI systems and robots. The complete source code of the framework is publicly available at https://github.com/snt-arg/robot_suite.

Figures (5)

• Figure 1: High-level overview of the proposed LLM-driven robotic control method, where a user interacts with the system through natural language, interpreted by an LLM to guide robot behavior via a modular control structure.
• Figure 2: The outline of the proposed system architecture. An LLM interprets natural language instructions from the human, interfacing with a behavior tree to coordinate modular plugins that control the robot’s actions.
• Figure 3: A detailed overview of the proposed system architecture, depicting the integration of LLM-based language understanding, behavior tree core, plugin, and driver modules. Arrow labels indicate the interaction category ($\Phi1$ to $\Phi6$) corresponding to evaluation scenarios described in §\ref{['sec_eval_setup']}.
• Figure 4: Structure of the sample behavior tree employed in the paper for system evaluation, containing the hierarchical arrangement of execution nodes to manage robotic behaviors.
• Figure 5: High-level state-flow diagram of the proposed end-to-end HRI system, highlighting command interpretation and execution states.