Towards Natural Language Environment: Understanding Seamless Natural-Language-Based Human-Multi-Robot Interactions

TL;DR

The paper addresses how humans can coordinate with multiple heterogeneous robots through natural language in a future home. It proposes the Natural Language Environment (NLE) as a conceptual design space and uses role-playing in virtual reality to explore interaction patterns, tensions, and design implications. The study identifies three key dimensions—task coordination dominance, robot autonomy, and robot personality—and derives design considerations to balance efficiency, trust, privacy, and user preferences. This work lays the groundwork for future, implementable language-driven multi-robot systems and informs designers about adaptable leadership, transparency, and user-centric customization in domestic settings.

Abstract

As multiple robots are expected to coexist in future households, natural language is increasingly envisioned as a primary medium for human-robot and robot-robot communication. This paper introduces the concept of a Natural Language Environment (NLE), defined as an interaction space in which humans and multiple heterogeneous robots coordinate primarily through natural language. Rather than proposing a deployable system, this work aims to explore the design space of such environments. We first synthesize prior work on language-based human-robot interaction to derive a preliminary design space for NLEs. We then conduct a role-playing study in virtual reality to investigate how people conceptualize, negotiate, and coordinate human-multi-robot interactions within this imagined environment. Based on qualitative and quantitative analysis, we refine the preliminary design space and derive design implications that highlight key tensions and opportunities around task coordination dominance, robot autonomy, and robot personality in Natural Language Environments.

Figures (10)

• Figure 1: Bird's-eye view of the apartment setting and interactable objects. a1) The initial setting before house cleaning. a2) An example of house setting after house cleaning. b1) Toys. b2) Dust. b3) Trash. b4) Clean tableware. b5) Dirty tableware. b6) Towel and cleaner.
• Figure 2: The four characters in the game: (a) the cleaning robot, (b) the first assistive robot, (c) the second assistive robot, and (d) the human.
• Figure 3: Some interactions between the characters. In (a), the two assistive robots work together to remove stains on a table. One robot applies cleaner on stains, while the other one wipes them off using a towel. (b) The two assistive robots adjust the coffee table collaboratively. (c) The human adjusts the heavy sofa by himself. (d)The red robot can not reach the high shelf and ask the human for help to put a clean bowl on the shelf.
• Figure 4: The VR views for the assistive robots, the cleaning robot, and the human. In the assistive robot view of (a), (a1) a SLAM map, (a2) an object detection simulation, and (a3) a to-do list are provided as generated by LLM. A hint is displayed when the assistive robot grabs an object, as shown in (b). (c) shows the cleaning robot view, where only dust is highlighted and labeled. The human view is depicted in (d), in which the human grabs two objects with both hands.
• Figure 5: The capabilities of the cleaning robot. In (a1), the cleaning robot looks under the sofa, which is difficult for other characters to see. The corresponding view of the robot is shown in (a2). (b) shows the cleaning robot vacuums the dust on the floor. (c) demonstrates the cleaning robot pushes out trash under the table.