Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community

TL;DR

The paper addresses how to broaden participation and advance research at the intersection of spoken language, robotics, and human–robot interaction by reporting a 2023 NSF workshop in SLIVAR. It proposes three white papers: educational resources, benchmarks and challenges, and integrating large language models with robots, each with concrete plans. It reviews existing work, outlines infrastructure such as a centralized resource repository, benchmark platforms, and virtual/real-world rigs, and discusses ethical, safety, and data-bias considerations. The work aims to foster community, reproducibility, and practical, safe, and effective human–robot dialogue with broad societal benefits.

Abstract

The ability to interact with machines using natural human language is becoming not just commonplace, but expected. The next step is not just text interfaces, but speech interfaces and not just with computers, but with all machines including robots. In this paper, we chronicle the recent history of this growing field of spoken dialogue with robots and offer the community three proposals, the first focused on education, the second on benchmarks, and the third on the modeling of language when it comes to spoken interaction with robots. The three proposals should act as white papers for any researcher to take and build upon.

Figures (3)

• Figure 1: Example of the Alexa Arena benchmark: user utterances and simulated robot responses, from Gao2023-ed
• Figure 2: The fields of robotics, hri, and sds have different, yet complementary goals, from Kennington2021-qh.
• Figure 3: Example of the ISABEL bot designed as part of the Amazon Alexa Taskbot Challenge 2023. ISABEL is the first multimodal bot that can interact with signers around the world add the role of LLM MalihePittsburgh2023.