Listen to the Unexpected: Self-Supervised Surprise Detection for Efficient Viewport Prediction

TL;DR

The paper addresses viewport prediction for 360-degree video by incorporating auditory surprise to anticipate off-screen gaze shifts. It introduces a self-supervised pipeline that projects spatial audio (First-Order Ambisonics) onto a spherical graph, processes it with an SE(3)-equivariant encoder, and uses a GRU to predict future states; surprise is defined as the prediction error, enabling temporal attention decay. The authors demonstrate that integrating audio surprise with visual cues reduces bitrate waste by up to 18% on AVTrack360 and improves QoE metrics like SSIM, illustrating practical gains for adaptive streaming. This work advances immersive media by bridging spatial audio perception and streaming optimization, with potential extensions to multi-modal fusion and user-specific attention modeling.

Abstract

Adaptive streaming of 360-degree video relies on viewport prediction to allocate bandwidth efficiently. Current approaches predominantly use visual saliency or historical gaze patterns, neglecting the role of spatial audio in guiding user attention. This paper presents a self-learning framework for detecting "surprising" auditory events -- moments that deviate from learned temporal expectations -- and demonstrates their utility for viewport prediction. The proposed architecture combines $SE(3)$-equivariant graph neural networks with recurrent temporal modeling, trained via a dual self-supervised objective. A key feature is the natural modeling of temporal attention decay: surprise is high at event onset but diminishes as the listener adapts. Experiments on the AVTrack360 dataset show that integrating audio surprise with visual cues reduces bitrate waste by up to 18% compared to visual-only methods.

Figures (5)

• Figure 1: The proposed Self-Learning Surprise Detection architecture. (A) Spherical Projection: Ambisonics audio is projected onto a dodecahedral graph. (B) SE(3)-Equivariant Encoder: A graph neural network extracts rotation-invariant scalars ($0e$) and equivariant vectors ($1o$). (C) Temporal Modeling: A GRU predicts future latent states $\hat{\mathbf{z}}_{t+k}$ from history $\mathbf{h}_t$. (D) Dual Loss: The model minimizes Contrastive (CPC) and Reconstruction losses simultaneously.
• Figure 2: Geometric Graph Construction. The Ambisonics audio field is sampled at the 20 vertices of an inscribed dodecahedron.
• Figure 3: Temporal dynamics of Surprise vs. Loudness. A sustained loud event (e.g., a siren) begins at $t=100$. (Blue) Raw loudness remains high for the duration. (Red) The proposed Surprise score spikes at onset ($t=100$) but decays exponentially as the GRU context $\mathbf{h}_t$ adapts to the new signal, reflecting human habituation.
• Figure 4: Audio-Visual Fusion Module. The system operates as a gated ensemble. The Visual Saliency Net proposes a standard heatmap $\mathbf{H}_v$. The Audio Encoder outputs a directional vector $\mathbf{v}_a$ and a scalar surprise $S(t)$. The surprise score is mapped to a mixing coefficient $\alpha \in [0, 1]$ via a sigmoid gate, blending the visual heatmap with the audio cue to form the final viewport probability $\mathbf{P}_{final}$.
• Figure 5: Qualitative comparison during an off-screen auditory event. The user is watching a street scene. (A) Ground Truth: An ambulance siren (red icon) activates behind the user (unseen). (B) Visual Saliency Baseline: The model predicts attention remains on the visible street center, failing to anticipate the turn. (C) Proposed Method: The audio surprise signal modifies the prediction heat map, correctly identifying the rear region as the target for the upcoming saccade.