Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models

Ammar N. Abbas, Csaba Beleznai

TL;DR

TalkWithMachines addresses interpretability in industrial robotics by combining Large Language Models and Vision-Language Models to translate a robot's perceptions, intentions, and constraints into human-readable explanations while enabling low-level language-driven control. The approach uses a modular framework where text and image prompts guide control commands, perception updates, and structure-aware planning grounded in URDF-defined robot geometry. Key contributions include language-based low-level control, verbalized machine states, perception augmented by image stacks, and URDF-informed structure awareness validated in simulated robotic-arm experiments involving safe zones, obstacle avoidance, stacking, and zone-based sorting. The results show improved trajectory generation, transparency, and safety-oriented decision making, pointing toward safer, more trustworthy human-robot collaboration in industry, with future work extending to real-world deployment and real-time perception.

Abstract

TalkWithMachines aims to enhance human-robot interaction by contributing to interpretable industrial robotic systems, especially for safety-critical applications. The presented paper investigates recent advancements in Large Language Models (LLMs) and Vision Language Models (VLMs), in combination with robotic perception and control. This integration allows robots to understand and execute commands given in natural language and to perceive their environment through visual and/or descriptive inputs. Moreover, translating the LLM's internal states and reasoning into text that humans can easily understand ensures that operators gain a clearer insight into the robot's current state and intentions, which is essential for effective and safe operation. Our paper outlines four LLM-assisted simulated robotic control workflows, which explore (i) low-level control, (ii) the generation of language-based feedback that describes the robot's internal states, (iii) the use of visual information as additional input, and (iv) the use of robot structure information for generating task plans and feedback, taking the robot's physical capabilities and limitations into account. The proposed concepts are presented in a set of experiments, along with a brief discussion. Project description, videos, and supplementary materials will be available on the project website: https://talk-machines.github.io.

TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models

TL;DR

TalkWithMachines addresses interpretability in industrial robotics by combining Large Language Models and Vision-Language Models to translate a robot's perceptions, intentions, and constraints into human-readable explanations while enabling low-level language-driven control. The approach uses a modular framework where text and image prompts guide control commands, perception updates, and structure-aware planning grounded in URDF-defined robot geometry. Key contributions include language-based low-level control, verbalized machine states, perception augmented by image stacks, and URDF-informed structure awareness validated in simulated robotic-arm experiments involving safe zones, obstacle avoidance, stacking, and zone-based sorting. The results show improved trajectory generation, transparency, and safety-oriented decision making, pointing toward safer, more trustworthy human-robot collaboration in industry, with future work extending to real-world deployment and real-time perception.

Abstract

TalkWithMachines aims to enhance human-robot interaction by contributing to interpretable industrial robotic systems, especially for safety-critical applications. The presented paper investigates recent advancements in Large Language Models (LLMs) and Vision Language Models (VLMs), in combination with robotic perception and control. This integration allows robots to understand and execute commands given in natural language and to perceive their environment through visual and/or descriptive inputs. Moreover, translating the LLM's internal states and reasoning into text that humans can easily understand ensures that operators gain a clearer insight into the robot's current state and intentions, which is essential for effective and safe operation. Our paper outlines four LLM-assisted simulated robotic control workflows, which explore (i) low-level control, (ii) the generation of language-based feedback that describes the robot's internal states, (iii) the use of visual information as additional input, and (iv) the use of robot structure information for generating task plans and feedback, taking the robot's physical capabilities and limitations into account. The proposed concepts are presented in a set of experiments, along with a brief discussion. Project description, videos, and supplementary materials will be available on the project website: https://talk-machines.github.io.

Paper Structure

This paper contains 32 sections, 2 equations, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. State-of-the-Art
  3. Contributions
  4. Methodology
  5. Framework
  6. Movement Descriptions and General Pattern Rules
  7. Prompt Structures
  8. Baseline Control Prompt Structure
  9. Context-Aware Perception: Verbalized Current Machine States
  10. Added Perception Through Visual Information
  11. Enhancing Awareness to Robot Structure
  12. Design of Experiments
  13. Language-Based Control
  14. Perception
  15. Experiments
  16. ...and 17 more sections

Figures (14)

  • Figure 1: Proposed framework: human-robot interaction interface from language to command and visual environment perception to human language.
  • Figure 2: Illustration of the axes of robot movement.
  • Figure 3: Prompt structures with incrementally added information, facilitating LLM-based reasoning and robot control. Blue indices refer to the experiments in Section \ref{['sec:results']}.
  • Figure 4: Design of experiments for control.
  • Figure 5: Design of experiments for perception through LLM.
  • ...and 9 more figures