Secure AI-Driven Super-Resolution for Real-Time Mixed Reality Applications
Mohammad Waquas Usmani, Sankalpa Timilsina, Michael Zink, Susmit Shannigrahi
TL;DR
The paper tackles the high bandwidth and motion-to-photon latency challenge in immersive MR streaming by integrating point-cloud downsampling at the origin, CP-ABE-based partial encryption, and client-side ML-driven super-resolution. It evaluates a secure SR distribution pipeline using LivingRoom and Office datasets, demonstrating near-linear reductions in bandwidth, latency, and crypto overhead as downsampling increases, while the Random Forest SR model achieves sub-millimeter geometric accuracy with modest inference times. The work uniquely combines cryptographic access control with ML-based upsampling, enabling scalable secure delivery of volumetric content and providing a practical framework for future end-to-end streaming with ABR and edge-accelerated inference. Overall, the approach offers a viable path to reduce data transfer and latency in MR applications without compromising access control, with clear avenues for real-time deployment and optimization.
Abstract
Immersive formats such as 360° and 6DoF point cloud videos require high bandwidth and low latency, posing challenges for real-time AR/VR streaming. This work focuses on reducing bandwidth consumption and encryption/decryption delay, two key contributors to overall latency. We design a system that downsamples point cloud content at the origin server and applies partial encryption. At the client, the content is decrypted and upscaled using an ML-based super-resolution model. Our evaluation demonstrates a nearly linear reduction in bandwidth/latency, and encryption/decryption overhead with lower downsampling resolutions, while the super-resolution model effectively reconstructs the original full-resolution point clouds with minimal error and modest inference time.