Audience Amplified: Virtual Audiences in Asynchronously Performed AR Theater

TL;DR

Audience Amplified addresses the challenge of delivering social presence in anytime, anywhere AR by training virtual audiences from real- world data using imitation learning. The approach combines location-based AR theater with an ML-trained NPC crowd, motion-matched avatars, and voice responses, evaluated through two pilots and a main within-subject study. Results show that virtual audiences increase perceived social engagement and time spent, though effects vary with introduction order and user height, revealing nuanced design tradeoffs. The work demonstrates a practical path toward asynchronous, socially connected AR performances and highlights considerations for avatar appearance, crowd layout, and environment scale in mobile AR settings.

Abstract

Audience reactions can considerably enhance live experiences; conversely, in anytime-anywhere augmented reality (AR) experiences, large crowds of people might not always be available to congregate. To get closer to simulating live events with large audiences, we created a mobile AR experience where users can wander around naturally and engage in AR theater with virtual audiences trained from real audiences using imitation learning. This allows us to carefully capture the essence of human imperfections and behavior in artificial intelligence (AI) audiences. The result is a novel mobile AR experience in which solitary AR users experience an augmented performance in a physical space with a virtual audience. Virtual dancers emerge from the surroundings, accompanied by a digitally simulated audience, to provide a community experience akin to immersive theater. In a pilot study, simulated human avatars were vastly preferred over just audience audio commentary. We subsequently engaged 20 participants as attendees of an AR dance performance, comparing a no-audience condition with a simulated audience of six onlookers. Through questionnaires and experience reports, we investigated user reactions and behavior. Our results demonstrate that the presence of virtual audience members caused attendees to perceive the performance as a social experience with increased interest and involvement in the event. On the other hand, for some attendees, the dance performances without the virtual audience evoked a stronger positive sentiment.

Figures (6)

• Figure 1: Users' perception of watching a dance performance with each of the three virtual audiences in the first pilot study: (a) A human figure made of basic geometric shapes represents the virtual audiences we named Icon Avatar. (b) A human figure avatar with 4 idle states represents virtual audiences. (c) A human figure avatar with 16 idle states, that enable the avatar to be more expressive and receptive, represents virtual audiences. (d) Accumulated preference ranking for each of the 3 avatars as virtual audiences (2 for the most preferred avatar, 1 for the next, and 0 for the least preferred, for a total of 20 participants). The avatar with 4 idle states was most preferred, followed by the basic geometric avatar, Icon Avatar.
• Figure 2: Design chart for Pilot Study 2, showing the four trial conditions and their components.
• Figure 3: The positioning and dispersion of virtual audiences and the differences between the ML and NPC models. (a) Demonstrates how audiences are positioned close to the dancers and are looking in the dancer's direction. (b) Illustrates how the crowd is dispersed and spaced out, with some people watching the dancers and others moving independently. (c) Accumulated preference rankings for each of the four experiment conditions in the second pilot study (with 3 points for the most preferred and 0 points for the least preferred, from a total of 10 participants).
• Figure 4: Trial completion time (left) and user experience ratings (right) for the two experiment conditions. Participants spent more time in the experience in the ML condition and also reported higher interest, involvement ("How much did the auditory aspects of the environment involve you?"), and investment("I was so involved that I felt that my actions could affect the activity"). For this and all following charts: Error bars represent 95% confidence interval. User experience ratings were on a 7-point Likert scale (7=Strongly agree/Very much, 1=Strongly disagree/Not at all).
• Figure 5: Presence (left) and user experience ratings (right) for the two groups of participants based on trial order (which condition they experienced first). Participants who experienced the Dance (no audience) condition first had a higher presence score, higher liveness (feeling of actually being in a live performance), and appreciated the virtual stage more.