Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Efficiency and Engagement in Scripted and LLM-Enhanced Human-Robot Interactions

Tim Schreiter, Jens V. Rüppel, Rishi Hazra, Andrey Rudenko, Martin Magnusson, Achim J. Lilienthal

TL;DR

The paper tackles whether Large Language Models can improve natural and robust human–robot interaction in dynamic industrial settings. It compares fully scripted interactions to LLM-enhanced responses during a representative pick-and-place task, using gaze tracking, motion capture, energy measurements, and standardized questionnaires to assess efficiency, engagement, and perception. Results show higher subjective engagement and perceived robot capability with LLMs, but objective performance remains similar for simple tasks, while LLMs incur additional energy costs and latency (~$2.5$ s). The findings suggest aligning interaction modality with task demands—PPS for straightforward, efficiency-driven tasks and LLM-based adaptation for more complex, dynamic scenarios—while highlighting design considerations around latency, energy, and deployment infrastructure, including potential benefits of local open-source models.

Abstract

To achieve natural and intuitive interaction with people, HRI frameworks combine a wide array of methods for human perception, intention communication, human-aware navigation and collaborative action. In practice, when encountering unpredictable behavior of people or unexpected states of the environment, these frameworks may lack the ability to dynamically recognize such states, adapt and recover to resume the interaction. Large Language Models (LLMs), owing to their advanced reasoning capabilities and context retention, present a promising solution for enhancing robot adaptability. This potential, however, may not directly translate to improved interaction metrics. This paper considers a representative interaction with an industrial robot involving approach, instruction, and object manipulation, implemented in two conditions: (1) fully scripted and (2) including LLM-enhanced responses. We use gaze tracking and questionnaires to measure the participants' task efficiency, engagement, and robot perception. The results indicate higher subjective ratings for the LLM condition, but objective metrics show that the scripted condition performs comparably, particularly in efficiency and focus during simple tasks. We also note that the scripted condition may have an edge over LLM-enhanced responses in terms of response latency and energy consumption, especially for trivial and repetitive interactions.

Evaluating Efficiency and Engagement in Scripted and LLM-Enhanced Human-Robot Interactions

TL;DR

The paper tackles whether Large Language Models can improve natural and robust human–robot interaction in dynamic industrial settings. It compares fully scripted interactions to LLM-enhanced responses during a representative pick-and-place task, using gaze tracking, motion capture, energy measurements, and standardized questionnaires to assess efficiency, engagement, and perception. Results show higher subjective engagement and perceived robot capability with LLMs, but objective performance remains similar for simple tasks, while LLMs incur additional energy costs and latency (~ s). The findings suggest aligning interaction modality with task demands—PPS for straightforward, efficiency-driven tasks and LLM-based adaptation for more complex, dynamic scenarios—while highlighting design considerations around latency, energy, and deployment infrastructure, including potential benefits of local open-source models.

Abstract

To achieve natural and intuitive interaction with people, HRI frameworks combine a wide array of methods for human perception, intention communication, human-aware navigation and collaborative action. In practice, when encountering unpredictable behavior of people or unexpected states of the environment, these frameworks may lack the ability to dynamically recognize such states, adapt and recover to resume the interaction. Large Language Models (LLMs), owing to their advanced reasoning capabilities and context retention, present a promising solution for enhancing robot adaptability. This potential, however, may not directly translate to improved interaction metrics. This paper considers a representative interaction with an industrial robot involving approach, instruction, and object manipulation, implemented in two conditions: (1) fully scripted and (2) including LLM-enhanced responses. We use gaze tracking and questionnaires to measure the participants' task efficiency, engagement, and robot perception. The results indicate higher subjective ratings for the LLM condition, but objective metrics show that the scripted condition performs comparably, particularly in efficiency and focus during simple tasks. We also note that the scripted condition may have an edge over LLM-enhanced responses in terms of response latency and energy consumption, especially for trivial and repetitive interactions.
Paper Structure (7 sections, 3 figures, 1 table)

This paper contains 7 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Experiment Design
  4. Participants
  5. Data Collection
  6. Results
  7. Discussion

Figures (3)

  • Figure 1: Setup for interacting with a NAO robot mounted on a forklift. The participant begins at the corridor entrance, the robot is located behind a wall. Interaction steps: Black: Carrying and placing a tin can; Red: Receiving instructions from the NAO robot; Green, Blue: Box delivery; Yellow: Interaction conclusion. Insets show egocentric eye-tracking with gaze sequences overlaid in brown.
  • Figure 2: Sequence plan of the experiment. The robot is either controlled by an LLM (in blue) or a pre-programmed schedule (PPS, in orange). Roman numbers I--IV denote the key interaction steps, color coded according to Figure \ref{['abb:expsetup']}.
  • Figure 3: Fixation heatmaps in the interaction and task execution phases for the LLM and PPS conditions. Subfigures (a) and (b) highlight differences in attention during robot interaction. Subfigures (c) and (d) highlight differences in visual attention during task execution.