Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Whispers of the Butterfly: A Research-through-Design Exploration of In-Situ Conversational AI Guidance in Large-Scale Outdoor MR Exhibitions

Dongyijie Primo Pan, Shuyue Li, Yawei Zhao, Junkun Long, Hao Li, Pan Hui

TL;DR

The paper addresses the challenge of scaling artwork interpretation in large-scale outdoor MR exhibitions by introducing Dream-Butterfly, a retrieval-grounded, multilingual conversational AI docent embodied as a non-humanoid companion. Using a Research-through-Design process, it implements Dream-Butterfly in a campus-scale outdoor MR exhibition and conducts an in-the-wild, between-subject study (N=24) to compare AI-first guiding with a primarily human-led tour while ensuring safety staff remain on-site. The study yields empirical evidence that making interpretation pull-based via AI-first guiding improves explanation access, immersion, and hedonic quality without increasing workload, while clarifying role boundaries and shifting pacing decisions to visitors. The findings offer transferable design implications for configuring mixed human–AI guiding roles, emphasizing language fidelity as a core component of presence in public, mobile MR contexts and highlighting the need for explicit role contracts and lightweight, safe interaction patterns.

Abstract

Large-scale outdoor mixed reality (MR) art exhibitions distribute curated virtual works across open public spaces, but interpretation rarely scales without turning exploration into a scripted tour. Through Research-through-Design, we created Dream-Butterfly, an in-situ conversational AI docent embodied as a small non-human companion that visitors summon for multilingual, exhibition-grounded explanations. We deployed Dream-Butterfly in a large-scale outdoor MR exhibition at a public university campus in southern China, and conducted an in-the-wild between-subject study (N=24) comparing a primarily human-led tour with an AI-led tour while keeping staff for safety in both conditions. Combining questionnaires and semi-structured interviews, we characterize how shifting the primary explanation channel reshapes explanation access, perceived responsiveness, immersion, and workload, and how visitors negotiate responsibility handoffs among staff, the AI guide, and themselves. We distill transferable design implications for configuring mixed human-AI guiding roles and embodying conversational agents in mobile, safety-constrained outdoor MR exhibitions.

Whispers of the Butterfly: A Research-through-Design Exploration of In-Situ Conversational AI Guidance in Large-Scale Outdoor MR Exhibitions

TL;DR

The paper addresses the challenge of scaling artwork interpretation in large-scale outdoor MR exhibitions by introducing Dream-Butterfly, a retrieval-grounded, multilingual conversational AI docent embodied as a non-humanoid companion. Using a Research-through-Design process, it implements Dream-Butterfly in a campus-scale outdoor MR exhibition and conducts an in-the-wild, between-subject study (N=24) to compare AI-first guiding with a primarily human-led tour while ensuring safety staff remain on-site. The study yields empirical evidence that making interpretation pull-based via AI-first guiding improves explanation access, immersion, and hedonic quality without increasing workload, while clarifying role boundaries and shifting pacing decisions to visitors. The findings offer transferable design implications for configuring mixed human–AI guiding roles, emphasizing language fidelity as a core component of presence in public, mobile MR contexts and highlighting the need for explicit role contracts and lightweight, safe interaction patterns.

Abstract

Large-scale outdoor mixed reality (MR) art exhibitions distribute curated virtual works across open public spaces, but interpretation rarely scales without turning exploration into a scripted tour. Through Research-through-Design, we created Dream-Butterfly, an in-situ conversational AI docent embodied as a small non-human companion that visitors summon for multilingual, exhibition-grounded explanations. We deployed Dream-Butterfly in a large-scale outdoor MR exhibition at a public university campus in southern China, and conducted an in-the-wild between-subject study (N=24) comparing a primarily human-led tour with an AI-led tour while keeping staff for safety in both conditions. Combining questionnaires and semi-structured interviews, we characterize how shifting the primary explanation channel reshapes explanation access, perceived responsiveness, immersion, and workload, and how visitors negotiate responsibility handoffs among staff, the AI guide, and themselves. We distill transferable design implications for configuring mixed human-AI guiding roles and embodying conversational agents in mobile, safety-constrained outdoor MR exhibitions.
Paper Structure (46 sections, 2 equations, 6 figures)

This paper contains 46 sections, 2 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Designing the Dream-Butterfly Guide
  4. RtD Framing: Scaling Interpretation in a Campus-Scale Outdoor MR Exhibition
  5. Design Strategy: Mixed Human--AI Roles Under Outdoor MR Constraints
  6. System Overview
  7. Embodiment: A Non-Humanoid Companion as Guide
  8. Why not a digital human.
  9. Why a butterfly.
  10. Form, material, and visual legibility.
  11. Interaction Ritual: Summon--Return--Dialogue
  12. Embodied Following Trajectory
  13. Grounded Multilingual Conversation via Retrieval-Augmented Generation
  14. Onboarding and In-Situ Usage Guidance
  15. Summary of the RtD Outcome
  16. ...and 31 more sections

Figures (6)

  • Figure 1: Field deployment overview in our campus-scale outdoor MR exhibition. Visitors explore the site wearing HMD-based MR headsets while Dream-Butterfly (top-center) follows nearby in an idle, non-intrusive mode until summoned.
  • Figure 2: A representative in-situ interactive label in the outdoor MR exhibition. Visitors can tap a work-anchored panel to read basic information while roaming; however, the label’s limited screen space and static presentation constrain the amount of interpretive detail and do not support open-ended, back-and-forth clarification.
  • Figure 3: Dream-Butterfly asset pipeline from modeling to final in-engine use: (a) C4D white-box geometry of the butterfly form; (b) rigging and skin-weight painting for wing deformation; (c) material/shader look-development and rendering in the game engine; (d) the final optimized butterfly asset with transparent background (shown here on white).
  • Figure 4: In-headset onboarding tutorial shown to every participant before entering the outdoor MR exhibition, providing step-by-step, live interaction training for both label reading and Dream-Butterfly conversation. (a) Language selection at startup. (b) Reading an artwork’s virtual label: aim the controller ray at a highlighted tag and press the trigger to open the label panel. (c) Asking the butterfly a question via voice: press-and-hold the grip button under the right middle finger to summon the AI guide and speak a question. (d) Real-time dialogue state: after activation, a microphone indicator confirms the system is listening while the butterfly stays near the user’s hand for turn-by-turn conversation.
  • Figure 5: Questionnaire outcomes by condition as experience traces. Left: Distributional comparisons (boxplots with individual points) for interpretive access, immersion/engagement, hedonic quality, and effort. Right: UEQ-S pragmatic and hedonic quality scores by condition (bars with error bars and individual points).
  • ...and 1 more figures