Effect of Adaptive Communication Support on LLM-powered Human-Robot Collaboration

TL;DR

This work addresses how LLM-powered robots should adapt their language-based feedback to human teammates across task complexities. The authors propose HRT-ML, a flexible framework with a low-frequency Coordinator and a high-frequency Manager that operate on a DAG-based subtask graph, supported by a Greedy Action Planner and four multimodal feedback modes. Through an Overcooked-based user study (varying map difficulty and feedback style), they show that proactive language feedback can boost trust, perceived intelligence, and collaboration efficiency, particularly as task complexity grows, while excessively frequent guidance can impede performance in simpler tasks. They also propose a simple adaptive principle that prescribes feedback style based on the comparison between task complexity, human capability, and LLM capability, guiding real-time communication strategies in human-robot teams. This work advances practical guidelines for deploying adaptive, language-driven collaboration in real-world HRI tasks and points to future work on real-time cognitive-load estimation and dynamic feedback adjustment.

Abstract

Effective human-robot collaboration requires robot to adopt their roles and levels of support based on human needs, task requirements, and complexity. Traditional human-robot teaming often relies on a pre-determined robot communication scheme, restricting teamwork adaptability in complex tasks. Leveraging strong communication capabilities of Large Language Models (LLMs), we propose a Human-Robot Teaming Framework with Multi-Modal Language feedback (HRT-ML), a framework designed to enhance human-robot interaction by adjusting the frequency and content of language-based feedback. HRT-ML framework includes two core modules: a Coordinator for high-level, low-frequency strategic guidance, and a Manager for subtask-specific, high-frequency instructions, enabling passive and active interactions with human teammates. To assess the impact of language feedback in collaborative scenarios, we conducted experiments in an enhanced Overcooked environment with varying levels of task complexity (easy, medium, hard) and feedback frequency (inactive, passive, active, superactive). Our results show that as task complexity increases relative to human capabilities, human teammates exhibited a stronger preference towards robotic agents that can offer frequent, proactive support. However, when task complexities exceed the LLM's capacity, noisy and inaccurate feedback from superactive robotic agents can instead hinder team performance, as it requires human teammates to increase their effort to interpret and respond to a large number of communications, with limited performance return. Our results offer a general principle for robotic agents to dynamically adjust their levels and frequencies of communications to work seamlessly with humans and achieve improved teaming performance.

Figures (8)

• Figure 1: (A) Cooking process to complete an order. (B) The designed human-AI collaboration interface (left: game layout, right: communication panel). The Blue hat agent and Green hat agent are operated by humans and robots, respectively. The red cross represents an example of the intermediate empty counter used to collaborate.
• Figure 2: Human-Robot Teaming Framework with Multi-Modal Feedback(HRT-ML). It contains two modules: the Coordinator and the Manager. The Coordinator designs overall coordination strategies using Directed Acyclic Graph (DAG) and dynamically add and delete nodes and edges through communication with humans. Three types of coordination—graph structure revision, graph node attribute revision, and adding temporary subtasks—are enclosed in a red dashed rectangle. A blue subtask node represents tasks that are ready for execution, a yellow node indicates tasks that are not yet ready, and a red node signifies emergency tasks. The action cost between two subtasks is indicated with numbers on the edges. The Manager interprets the graph-represented coordination strategy to allocate subtasks to humans and robots. It provides subtask-level instructions to guide humans toward the overall goal and updates the graph status upon the completion of each subtask.
• Figure 3: Overview of the human study procedure involving 16 participants, the study begins with a tutorial map to introduce game mechanics. Followed by the tutorial, the human players will then play on the easy, medium, and hard layouts with four agents with different language feedback. Post-session surveys were conducted to collect data on participant satisfaction, engagement, trust, and feedback.
• Figure 4: Human perceived robot intelligence level (blue bar) and trust level (red bar) represented on a seven-point Likert scale, ranging from "Very Unintelligent/Untrustworthy" to "Very Intelligent/Trustworthy".
• Figure 5: Game scores of all participants paired with different robotic agents across various layouts. The score distribution of each agent type, IFA, PFA, AFA, SFA, is represented by the blue, red, green, and yellow boxes, respectively.