Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IROSA: Interactive Robot Skill Adaptation using Natural Language

Markus Knauer, Samuel Bustamante, Thomas Eiband, Alin Albu-Schäffer, Freek Stulp, João Silvério

TL;DR

This work presents a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware.

Abstract

Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for robot skill adaptation from limited data. Combining these approaches holds significant promise for direct application to robotics, yet this combination has received limited attention, particularly for industrial deployment. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting robot skills without requiring fine-tuning or direct model-to-robot interaction. We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.

IROSA: Interactive Robot Skill Adaptation using Natural Language

TL;DR

This work presents a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware.

Abstract

Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for robot skill adaptation from limited data. Combining these approaches holds significant promise for direct application to robotics, yet this combination has received limited attention, particularly for industrial deployment. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting robot skills without requiring fine-tuning or direct model-to-robot interaction. We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.
Paper Structure (24 sections, 1 equation, 5 figures, 3 tables, 1 algorithm)

This paper contains 24 sections, 1 equation, 5 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Approach
  5. Problem Formulation
  6. Five-Step Workflow
  7. User Query
  8. Tool Selection
  9. Tool Parameterization
  10. Tool Execution
  11. Feedback and Iterative Refinement
  12. Complete Adaptation Workflow
  13. Evaluation
  14. Evaluation on real robot
  15. Experiment 1: Speed adaptation via natural language
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Overview of our approach Interactive RObot Skill Adaptation using natural language. Showing the interactive selection and parameterization of a tool by a LLM based on a user query leading to a skill adaptation via the used execution model. Some of the tools we are providing are shown. "Respond to User" is a general tool, whereas "Repulsion Point", "Via-Point Insertion" and "Speed Modulation" are specific tools to adapt KMPs.
  • Figure 2: Demonstration and prediction analysis for the pick-and-insert task. (Top) Trajectory in robot frame. (Bottom) Demonstrations and KMP model predictions with uncertainty visualization.
  • Figure 3: Speed adaptation results showing temporal trajectory modification through natural language commands. The plot shows the adapted trajectories following command "slow down between box and station," demonstrating precise temporal control while preserving spatial trajectory characteristics.
  • Figure 4: Trajectory adaptation through natural language command showing (top) experimental setup with new object (camera) placement and (bottom) resulting KMP trajectory modification with via-point insertion to reach the new object while maintaining task completion.
  • Figure 5: Obstacle avoidance through natural language command showing (top) experimental workspace with introduced obstacle and (bottom) KMP trajectory modification using repulsion points. The system generates collision-free paths while maintaining the overall task structure and successfully completing the pick-and-insert operation.