Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz

TL;DR

This paper addresses the challenge of prototyping multimodal, real-time GenAI agents by introducing a user-centered prototyping pipeline that combines Counterfactual Video Replay Prompting with Hybrid Wizard-of-Oz methods. Through the SocraBot case study, the authors demonstrate how immersive, experiential evaluation and live human-in-the-loop interventions can iteratively refine prompting strategies, decomposing complex prompts into specialized modules to align outputs with unfolding user activity. They further contribute an open-source Counterfactual Replay Toolkit that enables rapid prototyping, evaluation, and integration with hybrid WoZ workflows across domains. The work highlights practical benefits for tameing design-time uncertainty in Proactive, context-aware design assistants and outlines future directions for integrating replay-based evaluation with existing prototyping and evaluation ecosystems. Together, these contributions offer a concrete, designerly pathway to develop and evaluate sophisticated multimodal AI agents that assist users in complex, real-world workflows.

Abstract

Recent advancements in multimodal generative AI (GenAI) enable the creation of personal context-aware real-time agents that, for example, can augment user workflows by following their on-screen activities and providing contextual assistance. However, prototyping such experiences is challenging, especially when supporting people with domain-specific tasks using real-time inputs such as speech and screen recordings. While prototyping an LLM-based proactive support agent system, we found that existing prototyping and evaluation methods were insufficient to anticipate the nuanced situational complexity and contextual immediacy required. To overcome these challenges, we explored a novel user-centered prototyping approach that combines counterfactual video replay prompting and hybrid Wizard-of-Oz methods to iteratively design and refine agent behaviors. This paper discusses our prototyping experiences, highlighting successes and limitations, and offers a practical guide and an open-source toolkit for UX designers, HCI researchers, and AI toolmakers to build more user-centered and context-aware multimodal agents.

Figures (4)

• Figure 1: SocraBot's front-end user interface. (1) While a designer works on a mechanical design task in Autodesk Fusion 360 (2) SocraBot follows the user's verbalizations and screen actions and proactively sends support messages that appear in the overlay GUI window; new messages will also be automatically read as a synthesized voice message to the user.
• Figure 2: Process diagram illustrating the sequence of our prototyping steps, employed evaluation approaches, and reflections, alongside revealed prompt issues and improvements. The bottom row shows the resulting evolution of our prompt strategy over the course of the prototyping process.
• Figure 3: Interface of the Counterfactual Replay Prompting tool used during prototyping; using the video player (1), one can explore different situations of recorded user sessions along with the synchronized transcription (2), and then generate a model response for a specific situation (3); and review (4) and evaluate (5) the generated message. Note: This interface differs from the open-source version.
• Figure 4: The Hybrid Wizard-of-Oz setup; (1) Wizards can follow along with user actions in the video call app; (2) using the wizard interface, they can trigger various model inference calls such as (3) task phase classification and (4) message generation; (5) wizards can optionally send generated messages to the user's SocraBot interface; (6) all generated messages can be reviewed and rated in the message history. Note: This interface differs from the open-source version.