The Stage Comes to You: A Real-Time Tele-Immersive System with 3D Point Clouds and Vibrotactile Feedback

TL;DR

This work tackles real-time, stage-scale tele-immersion by integrating low-latency 3D streaming with vibrotactile feedback for distant audiences. The authors introduce Dynamic Point-Cloud Space Transmission (DPCS) for robust, multi-unit LiDAR/RGB-D capture and a view-dependent depth bias to minimize seams, paired with Vibrotactile-Sound Field Presentation (VSFP) that uses shoe-mounted accelerometers and a propagation-aware floor to deliver synchronized footstep vibrations. In a public Expo 2025 trial spanning 20 km, the system achieved end-to-end latency of 81.3 ms and streamed 3.5 Gbps of 1.5 million-point frames across seven capture units, with 128 floor actuators shaping the haptic field. The results demonstrate the first stage-scale integration of real-time 3D and haptic transmission for tele-immersive performances, enabling natural two-way interaction and broad audience engagement in large spaces.

Abstract

We present a low-latency tele-immersive entertainment system that streams 3D point clouds and performers' footstep vibrations, creating the sense that the stage is present. Moving performers and their surroundings are captured as dynamic point clouds under rapidly changing lighting, then processed, transmitted, and rendered within a total latency of less than 100 ms. Under high ambient noise, footstep vibrations are sensed by wearable accelerometers. Real-time visual and haptic streams are delivered to a remote venue, where a large 3D LED wall and a vibration-efficient haptic floor envelop dozens of spectators. A public trial at Expo 2025 linked sites 20 km apart: visitors watched a live dance show and conversed with performers without noticeable delay.