InstructRobot: A Model-Free Framework for Mapping Natural Language Instructions into Robot Motion

TL;DR

The paper addresses the problem of translating natural language instructions into robot actions under data scarcity and high-DoF constraints. It proposes InstructRobot, a dataset-free, model-free reinforcement learning framework that jointly learns language representations and inverse kinematics without requiring prior IK models. The approach uses a PPO-based policy with a Transformer-based Language System and multimodal Perceptual System, demonstrating results on a 26-DoF NAO robot across single- and multi-instruction tasks, with open-source code available. This work advances human-robot interaction by enabling robust, instruction-guided manipulation in realistic, data-scarce domains and highlights avenues for future improvements through reward design and language-model integration.

Abstract

The ability to communicate with robots using natural language is a significant step forward in human-robot interaction. However, accurately translating verbal commands into physical actions is promising, but still presents challenges. Current approaches require large datasets to train the models and are limited to robots with a maximum of 6 degrees of freedom. To address these issues, we propose a framework called InstructRobot that maps natural language instructions into robot motion without requiring the construction of large datasets or prior knowledge of the robot's kinematics model. InstructRobot employs a reinforcement learning algorithm that enables joint learning of language representations and inverse kinematics model, simplifying the entire learning process. The proposed framework is validated using a complex robot with 26 revolute joints in object manipulation tasks, demonstrating its robustness and adaptability in realistic environments. The framework can be applied to any task or domain where datasets are scarce and difficult to create, making it an intuitive and accessible solution to the challenges of training robots using linguistic communication. Open source code for the InstructRobot framework and experiments can be accessed at https://github.com/icleveston/InstructRobot.

Figures (8)

• Figure 1: An illustration of the InstructRobot framework. Our framework enables natural language instructions to be mapped into robot motion without requiring the creation of datasets or knowledge of inverse kinematic models.
• Figure 2: The InstructRobot framework comprises two main blocks: environment and agent. The Environment block was designed to simplify the process of generating task instructions and rewards and is composed mainly of the Instructional Set, Reward Generator, and Robot modules. For every episode, an instruction $i_t$ is randomly selected from the Instructional Set and becomes the active instruction of the environment, providing a designed reward function to evaluate the robot's actions. The Agent block comprises the Language System, Perceptual System, Alignment, and Actor and Critic modules. The agent receives a state $\textbf{s}_{t+1}$ that comprises the active instruction $i_t$ and the perceptual information $\textbf{p}_t$ from the environment, processes it in its subsystems and acts in the environment by sending an action $\textbf{a}_t$, receiving a reward $r_t$. In this process, the agent learns a policy $\pi(\textbf{a}|\textbf{s})$ that maps the instruction into robot motion.
• Figure 3: Mean episodic return for our single-instruction experiment. The agent could learn a policy to complete the task, achieving a mean return of $2.4$.
• Figure 4: Policy execution for our single-instruction experiment. The agent's goal was to touch only the blue cube. Note that the agent can move its arms and fingers to the correct cube and keep them in that position for the entire trajectory.
• Figure 5: Mean episodic return for the multi-instructions experiment. The agent was trained for 2.2 million steps, achieving a mean episodic return of 5.