VoLUT: Efficient Volumetric streaming enhanced by LUT-based super-resolution

TL;DR

VoLUT tackles the bandwidth challenge of volumetric video by introducing a LUT-based 3D point-cloud super-resolution pipeline that splits SR into fast dilated interpolation and LUT-driven refinement. It couples this SR with a continuous adaptive bitrate streaming framework optimized by Model Predictive Control to achieve arbitrary downsampling while preserving QoE. The approach delivers real-time performance on mobile-like hardware, reduces data usage by up to ~70%, and outperforms state-of-the-art SR-based systems in QoE, thanks to efficient LUTs and a robust interpolation scheme. This work offers a practical pathway for scalable, high-quality 6DoF volumetric streaming on consumer devices.

Abstract

3D volumetric video provides immersive experience and is gaining traction in digital media. Despite its rising popularity, the streaming of volumetric video content poses significant challenges due to the high data bandwidth requirement. A natural approach to mitigate the bandwidth issue is to reduce the volumetric video's data rate by downsampling the content prior to transmission. The video can then be upsampled at the receiver's end using a super-resolution (SR) algorithm to reconstruct the high-resolution details. While super-resolution techniques have been extensively explored and advanced for 2D video content, there is limited work on SR algorithms tailored for volumetric videos. To address this gap and the growing need for efficient volumetric video streaming, we have developed VoLUT with a new SR algorithm specifically designed for volumetric content. Our algorithm uniquely harnesses the power of lookup tables (LUTs) to facilitate the efficient and accurate upscaling of low-resolution volumetric data. The use of LUTs enables our algorithm to quickly reference precomputed high-resolution values, thereby significantly reducing the computational complexity and time required for upscaling. We further apply adaptive video bit rate algorithm (ABR) to dynamically determine the downsampling rate according to the network condition and stream the selected video rate to the receiver. Compared to related work, VoLUT is the first to enable high-quality 3D SR on commodity mobile devices at line-rate. Our evaluation shows VoLUT can reduce bandwidth usage by 70% , boost QoE by 36.7% for volumetric video streaming and achieve 3D SR speed-up with no quality compromise.

Figures (18)

• Figure 1: DL-based Point Cloud Super-resolution: Direct vs. Two-stage.
• Figure 2: The System Architecture of $\sf\small{VoLUT}$.
• Figure 3: The pipeline of two-stage Super-resolution with LUT refinement
• Figure 4: Qualitative upsampling results (from left to right): Groundtruth, Interpolation with dilation, Naive knn-based interpolation. Our method achieves more uniform point distribution while preserving geometric details.
• Figure 5: Interpolation with and without dilation. Receptive Field size = $k \times$ dilation. The dilated approach significantly improves point distribution uniformity and surface coverage.