Foveated Compression for Immersive Telepresence Visualization

TL;DR

This work addresses bandwidth bottlenecks in immersive telepresence video by introducing a real-time foveated compression scheme that modulates HEVC quantization using eye-tracking data. The method computes a spatial QP delta map with a Gaussian mask centered on the gaze projection and applies it to 32×32 macroblocks, with an optional foveated warping option, integrated into the NimbRo Avatar system. Evaluations show bandwidth reductions approaching about one-third of baseline while maintaining immersion, with measured latency around $50~\mathrm{ms}$ in wired setups and discussion of network-dependent effects. The approach provides a codec-friendly, scalable path to high-fidelity telepresence in bandwidth-limited environments and opens avenues for further improvements such as gaze-prediction during saccades.

Abstract

Immersive televisualization is important both for telepresence and teleoperation, but resolution and fidelity are often limited by communication bandwidth constraints. We propose a lightweight method for foveated compression of immersive televisualization video streams that can be easily integrated with common video codecs, reducing the required bandwidth if eye tracking data is available. Specifically, we show how to spatially adjust the Quantization Parameter of modern block-based video codecs in a adaptive way based on eye tracking information. The foveal region is transmitted with high fidelity while quality is reduced in the peripheral region, saving bandwidth. We integrate our method with the NimbRo avatar system, which won the ANA Avatar XPRIZE competition. Our experiments show that bandwidth can be reduced to a third without sacrificing immersion. We analyze transmission fidelity with qualitative examples and report quantitative results.

Figures (9)

• Figure 1: Overview of our method. We provide immersive visualization of the robot's environment to the operator by streaming stereo images to the operator's HMD. For transmission over a potentially bandwidth-limited network, the video stream is encoded using HEVC. We propose to utilize eye tracking information to modulate HEVC's Quantization Parameter (QP) spatially---decreasing quality (and thus bandwidth usage) in areas that are not in the focus of the operator.
• Figure 2: Eye tracking pipeline. We mount additional eye cameras into an off-the-shelf HMD (Valve Index). The captured eye images are processed by a lightweight hourglass network to heatmaps describing an eye keypoint. Finally, the detected keypoints are transformed into the eye coordinate system and averaged to determine the gaze direction.
• Figure 3: Avatar robot head & Spherical rendering. (a) The NimbRo avatar robot head with a 6-DoF neck and stereo cameras. (b) Spherical rendering example. We show only one camera $C$ of the stereo pair, the other is processed analogously. The robot camera is shown in white with its very wide FoV. The corresponding VR camera $V$, which renders the view displayed to the operator, is shown in green. The camera image is projected onto the sphere with radius $r$, and then back into the VR camera. Adapted from schwarz2021low.
• Figure 4: Exemplary QP modulation mask for an image of size 2472$\times$2178, resulting in a mask size of 78$\times$69. White corresponds to a high QP change (yielding higher compression) while black indicates no QP change.
• Figure 5: Foveated warping example. The input image (a) is warped so that pixel density, visualized in a polar fixation-centric coordinate system (e) follows the cutoff frequency of the human vision system (d). However, this wastes space in the corners. By restricting the warping to individual axes, the entire space can be utilized (c), however this introduces additional modes in pixel density (f). In the lower plots, darker color indicates higher frequency or density.