ARCADE: An Augmented Reality Display Environment for Multimodal Interaction with Conversational Agents

TL;DR

ARCADE addresses the challenge of natural, ubiquitous interaction with embodied conversational agents by embedding them in the real environment through a borderless, human-scale AR display. It combines a high-luminosity, transparent back-projected surface with straightforward software integration to host conventional dialogue systems and multimodal content. The authors demonstrate two agent prototypes—one for multimodal chit-chat driven by ChatGPT and another for explainable AI via Athena—highlighting the system's versatility. The work contributes hardware design, integration methodology, and practical demonstrations that enable seamless porting of existing agents to an in-room AR display, with future directions including sensor augmentation and user studies to assess usability and impact.

Abstract

Making the interaction with embodied conversational agents accessible in a ubiquitous and natural manner is not only a question of the underlying software but also brings challenges in terms of the technical system that is used to display them. To this end, we present our spatial augmented reality system ARCADE, which can be utilized like a conventional monitor for displaying virtual agents as well as additional content. With its optical-see-through display, ARCADE creates the illusion of the agent being in the room similarly to a human. The applicability of our system is demonstrated in two different dialogue scenarios, which are included in the video accompanying this paper at https://youtu.be/9nH4c4Q-ooE.

Figures (1)

• Figure 1: ARCADE from a user's perspective.