Leveraging Large Language Models in Human-Robot Interaction: A Critical Analysis of Potential and Pitfalls

TL;DR

A meta-study of more than 250 papers exploring major robots in HRI research and significant applications of SARs outlines a pathway for the responsible and effective adoption of LLM or VLM into SARs, and offers caution regarding this deployment.

Abstract

The emergence of large language models (LLM) and, consequently, vision language models (VLM) has ignited new imaginations among robotics researchers. At this point, the range of applications to which LLM and VLM can be applied in human-robot interaction (HRI), particularly socially assistive robots (SARs), is unchartered territory. However, LLM and VLM present unprecedented opportunities and challenges for SAR integration. We aim to illuminate the opportunities and challenges when roboticists deploy LLM and VLM in SARs. First, we conducted a meta-study of more than 250 papers exploring 1) major robots in HRI research and 2) significant applications of SARs, emphasizing education, healthcare, and entertainment while addressing 3) societal norms and issues like trust, bias, and ethics that the robot developers must address. Then, we identified 4) critical components of a robot that LLM or VLM can replace while addressing the 5) benefits of integrating LLM into robot designs and the 6) risks involved. Finally, we outline a pathway for the responsible and effective adoption of LLM or VLM into SARs, and we close our discussion by offering caution regarding this deployment.

Figures (7)

• Figure 1: HRI papers in our meta-study included participants. We summarize the age and gender distribution of participant information from the papers. In the age distribution, we define children as those under 18 years old, while adults are 18 years and above. For gender distribution, we summarized female, male, and other.
• Figure 2: The papers in our meta-study often included participants. We summarize the distribution of the size of participant groups inside the HRI papers. Many studies included more than 100 participants.
• Figure 3: Robots in HRI research. The horizontal axis indicates the number of times each robot appeared during our study and the percentage inside parenthesis.
• Figure 4: Examples of socially assistive robots (SARs) encountered in our study. The robots are deployed in healthcare, education, entertainment, and hospitality&services applications. (Most images are taken from the ABOT Database).
• Figure 5: Key components of the Sophia robot.