Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies

TL;DR

This work introduces GIDEA, a generative-agent based simulation platform that models human–assistant interactions to replicate ten published studies. By integrating modular components (Interaction Knowledge, Context Setup, Assistant Agent–Avatar Interaction) and validating across multiple language models, GIDEA demonstrates high semantic fidelity and practical utility for scalable, cost-effective early-stage design. The study provides cross-model evidence of generalizability and employs rigorous bias mitigation and data leakage checks, establishing a model-agnostic approach for HCI research workflows. While limitations such as multimodal inputs and long-term dynamics are acknowledged, GIDEA offers a traceable, reusable framework to explore personalization, proactivity, interruptibility, and user control in intelligent assistants. The platform and results are open resources, accelerating iterative design and evaluation in real-world human–system interaction contexts.

Abstract

Designing and evaluating personalized and proactive assistant agents remains challenging due to the time, cost, and ethical concerns associated with human-in-the-loop experimentation. Existing Human-Computer Interaction (HCI) methods often require extensive physical setup and human participation, which introduces privacy concerns and limits scalability. Simulated environments offer a partial solution but are typically constrained by rule-based scenarios and still depend heavily on human input to guide interactions and interpret results. Recent advances in large language models (LLMs) have introduced the possibility of generative agents that can simulate realistic human behavior, reasoning, and social dynamics. However, their effectiveness in modeling human-assistant interactions remains largely unexplored. To address this gap, we present a generative agent-based simulation platform designed to simulate human-assistant interactions. We identify ten prior studies on assistant agents that span different aspects of interaction design and replicate these studies using our simulation platform. Our results show that fully simulated experiments using generative agents can approximate key aspects of human-assistant interactions. Based on these simulations, we are able to replicate the core conclusions of the original studies. Our work provides a scalable and cost-effective approach for studying assistant agent design without requiring live human subjects. Additional resources and project materials are available at https://dash-gidea.github.io/

Figures (8)

• Figure 1: (a) Existing HCI studies follow a resource-intensive workflow from design to deployment and data collection. (b) Core components span evaluation metrics, assistant interfaces, participant traits, and physical environments. (c) The proposed LLM-driven framework enables efficient simulation through automated assistant agent and avatar interaction modeling.
• Figure 2: Simulation workflow of GIDEA. In the initialization phase, researchers provide interaction knowledge, avatar profiles, environment configurations, and memory setup. During runtime, the Assistant Agent–Avatar Interaction module executes iterative study rounds, generating dialogues, updating memory and environment states, and producing simulation results.
• Figure 3: Semantic similarity scores comparing simulated and original study responses for each research question (RQ) across 10 case studies. The red line shows the overall average similarity (0.85). Study-specific average similarities are in the legend.
• Figure 4: Average semantic similarity scores grouped by (a) study theme—User Control, Interruptibility, Personalization, and Proactivity—and (b) study mode—Wizard-of-Oz (WoZ), Storyboard, and Interview.
• Figure 5: Personalization and Social Framing.