Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Concert Interaction Translation: Augmenting VR Live Concert Experience using Chat-Driven Artificial Collective Reactions

Sebin Lee, Yeonho Cho, Jungjin Lee

TL;DR

The paper addresses limited social experience in VR live concerts by leveraging real-time live-stream chat to generate collective artificial audience reactions. It introduces the Concert Interaction Translation (CIT) system, which analyzes 68 offline concert videos to derive movement and sound reaction templates and maps real-time Korean chat emotions to avatar motions, cheering, and singalong cues in a VR venue. A proof-of-concept prototype implemented in Unity with a FastAPI server and KOTE-based emotion classification demonstrates that chat-driven reactions elevate immersion and co-presence compared with a Baseline that relies on acoustic features alone, as shown in a user study with n=48. The work highlights the potential of cross-platform audience interaction to enrich computer-mediated concerts and outlines avenues for broader validation and automation of reactions.

Abstract

Computer-mediated concerts can be enjoyed on various devices, from desktop and mobile to VR devices, often supporting multiple devices simultaneously. However, due to the limited accessibility of VR devices, relatively small audience members tend to congregate in VR venues, resulting in diminished unique social experiences. To address this gap and enrich VR concert experiences, we present a novel approach that leverages non-VR user interaction data, specifically chat from audiences watching the same content on a live-streaming platform. Based on an analysis of audience reactions in offline concerts, we designed and prototyped a concert interaction translation system that extracts the level of engagement and emotions from chats and translates them to collective movements, cheers, and singalongs of virtual audience avatars in a VR venue. Our user study (n=48) demonstrates that our system, which combines both movement and audio reactions, significantly enhances the sense of immersion and co-presence than the previous method.

Concert Interaction Translation: Augmenting VR Live Concert Experience using Chat-Driven Artificial Collective Reactions

TL;DR

The paper addresses limited social experience in VR live concerts by leveraging real-time live-stream chat to generate collective artificial audience reactions. It introduces the Concert Interaction Translation (CIT) system, which analyzes 68 offline concert videos to derive movement and sound reaction templates and maps real-time Korean chat emotions to avatar motions, cheering, and singalong cues in a VR venue. A proof-of-concept prototype implemented in Unity with a FastAPI server and KOTE-based emotion classification demonstrates that chat-driven reactions elevate immersion and co-presence compared with a Baseline that relies on acoustic features alone, as shown in a user study with n=48. The work highlights the potential of cross-platform audience interaction to enrich computer-mediated concerts and outlines avenues for broader validation and automation of reactions.

Abstract

Computer-mediated concerts can be enjoyed on various devices, from desktop and mobile to VR devices, often supporting multiple devices simultaneously. However, due to the limited accessibility of VR devices, relatively small audience members tend to congregate in VR venues, resulting in diminished unique social experiences. To address this gap and enrich VR concert experiences, we present a novel approach that leverages non-VR user interaction data, specifically chat from audiences watching the same content on a live-streaming platform. Based on an analysis of audience reactions in offline concerts, we designed and prototyped a concert interaction translation system that extracts the level of engagement and emotions from chats and translates them to collective movements, cheers, and singalongs of virtual audience avatars in a VR venue. Our user study (n=48) demonstrates that our system, which combines both movement and audio reactions, significantly enhances the sense of immersion and co-presence than the previous method.

Paper Structure

This paper contains 17 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Analysis of Audience Reactions in Offline Concerts
  3. Movement Reactions
  4. Sound Reactions
  5. Cheering Sounds
  6. Singalong Sounds
  7. Concert Interaction Translation
  8. Triggering Movement Reactions
  9. Triggering Cheering Sounds
  10. Triggering Singalong Sounds
  11. Implementation of the Prototype System
  12. User Study
  13. Study Design
  14. Procedure
  15. Result
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Description of observed movement reactions
  • Figure 2: Overview of the proposed system, Concert Interaction Translation (CIT)
  • Figure 3: Images captured from the CIT prototype (left: shaking arms back and forth to a fast-tempo song, center: clapping hands during a talk session, right: clapping hands to a slow-tempo song)
  • Figure 4: The results of the analysis on overall sickness, presence, immersion, and co-presence according to four scenarios (S1-S4). (signif. level: * $p<.05$, ** $p<.01$)