From Temporal to Spatial: Designing Spatialized Interactions with Segmented-audios in Immersive Environments for Active Engagement with Performing Arts Intangible Cultural Heritage

TL;DR

This paper tackles the challenge of engaging audiences with auditory Intangible Cultural Heritage such as Peking Opera by transforming passive listening into active, spatial exploration through a Spatial Interaction-based Segmented-Audio (SISA) VR system. It employs a three-phase methodology—co-design with stakeholders, iterative prototyping, and user testing—to derive design rationales, validate technical feasibility, and reveal user interaction patterns. The final findings identify Progressive and Adaptive exploration modes and demonstrate that spatially organized audio segments, clustered via MFCC features and t-SNE, can enhance engagement and learning while preserving cultural authenticity. The work offers a replicable design framework for digital ICH preservation and presents practical implications for immersive exhibits, educational tools, and interactive media that aim to keep traditional performing arts vibrant in the digital age.

Abstract

Performance artforms like Peking opera face transmission challenges due to the extensive passive listening required to understand their nuance. To create engaging forms of experiencing auditory Intangible Cultural Heritage (ICH), we designed a spatial interaction-based segmented-audio (SISA) Virtual Reality system that transforms passive ICH experiences into active ones. We undertook: (1) a co-design workshop with seven stakeholders to establish design requirements, (2) prototyping with five participants to validate design elements, and (3) user testing with 16 participants exploring Peking Opera. We designed transformations of temporal music into spatial interactions by cutting sounds into short audio segments, applying t-SNE algorithm to cluster audio segments spatially. Users navigate through these sounds by their similarity in audio property. Analysis revealed two distinct interaction patterns (Progressive and Adaptive), and demonstrated SISA's efficacy in facilitating active auditory ICH engagement. Our work illuminates the design process for enriching traditional performance artform using spatially-tuned forms of listening.

Figures (7)

• Figure 1: A detailed break down of our project phases.
• Figure 2: SISA Audio Processing Workflow consists of the following steps: (1) Audio Separation and Segmentation, (2) Feature Extraction, (3) Dimensionality Reduction. In this illustration, the waveform, spectrograms, and the heat map are derived from a short testing clip of Kunqu Opera.
• Figure 3: SISA Spatial Mapping Workflow. (Left) 2D Visualization of t-SNE Result with DBSCAN Labeling and Manual Annotation. For the selected Kunqu Opera music, t-SNE effectively grouped similar features together. Using DBSCAN, six distinct clusters were identified, differentiating between instruments and character roles, and whether the performance involved singing or reciting. (Right) 3D Mapping of t-SNE Result and its top view. Using a cylindrical coordinate system, the 2D t-SNE results were transformed into a 3D space, ensuring that when the user is positioned at the center, they are equidistant from all points.
• Figure 4: SISA Visual Environment Generation Workflow. We used Skybox AI to generate a visual environment of the ICH scene. The process begins with extracting keywords from UNESCO genre descriptions, followed by combining these keywords into a Skybox-specific prompt. Skybox AI then uses this prompt to create the 360° panoramic image, resulting in a detailed visual representation of the ICH site. (A) Panoramic image of Kun Qu Opera with the absence of people. (B) Panoramic image of Kun Qu Opera with the presence of people.
• Figure 5: System Overview of the SISA illustrated with Peking Opera genre. The audio processing takes the ICH music as input and perform t-SNE algorithms after segmentation and MFCC feature extraction. The 2D result is then translated into 3D mapping and with the generated virtual environment for a culturally relevant immersive environment. Both components feed into the SISA system giving us a VR experience that enable to spatially explore the ICH performance arts.