Meaning over Motion: A Semantic-First Approach to 360° Viewport Prediction

TL;DR

The paper tackles bandwidth- and stall-heavy 360° video delivery by reframing viewport prediction around semantic intent rather than purely kinematic cues. It proposes Semantically-Adaptive Conformal Tiling with Associative Lookahead, implemented via architectural inversion that offloads heavy semantic processing to a server and uses a lightweight client controller guided by a Mondrian Conformal Prediction framework. A Semantic Tile Map and an Association Graph provide probabilistic semantic foresight, enabling a Dual-Set (Foveal Maintenance and Distal Lookahead) prediction strategy with dynamic margins and adaptive risk budgeting. Empirical results on the 360-AV-HM dataset show significant improvements in stall reduction ($\ge$20%) and bandwidth savings ($\ge$18%) over state-of-the-art baselines, with personalization and associative lookahead identified as key drivers of performance and feasibility for real-time deployment.

Abstract

Ultra-high-resolution 360-degree video streaming is severely constrained by the massive bandwidth required to deliver immersive experiences. Current viewport prediction techniques predominately rely on kinematics or low-level visual saliency, treating users as passive physical objects governed by inertia. This theoretical limitation leads to the "Saccade Trap" -- a critical failure mode where predictors fail to anticipate rapid, meaning-driven shifts in attention, causing rebuffering stalls exactly when user engagement is highest. To resolve this, we propose Semantically-Adaptive Conformal Tiling with Associative Lookahead, a novel framework that integrates cognitive intent into network control. Unlike "one-size-fits-all" approaches, our method utilizes an architectural inversion strategy: heavy semantic reasoning is offloaded to the server to generate lightweight association graphs, which guide a low-latency client-side controller. We construct a personalized Multi-Modal Prediction Set that dynamically tightens safety margins during stable fixation to maximize efficiency, while simultaneously pre-fetching non-adjacent tiles containing semantically linked objects (Associative Lookahead). This mechanism effectively converts the "calm" of fixation into a preparation phase for the next interaction. Trace-driven evaluation on the 360-AV-HM dataset demonstrates that this approach successfully mitigates the Saccade Trap, reducing stall duration by $\ge$ 20% and lowering effective bandwidth consumption by $\ge$ 18% compared to state-of-the-art trajectory-based baselines.

Figures (5)

• Figure 1: Illustration of the "Saccade Trap." Kinematic predictors (dashed line) rely on inertia and fail to anticipate rapid saccades (solid line) driven by semantic interest. Our approach leverages the associative link (green arrow) to pre-fetch the destination before motion begins.
• Figure 2: Proposed System Architecture. High-latency semantic segmentation and association mining are offloaded to the server (offline). The client (online) executes a lightweight kinematic transformer modulated by server-generated semantic metadata to ensure real-time responsiveness.
• Figure 3: Construction of the Multi-Modal Prediction Set. Unlike standard concentric buffers (dotted line), our policy constructs a union of a tight Foveal Maintenance Set (Green) and a distal Associative Lookahead Set (Blue), optimizing bandwidth usage during semantic transitions.
• Figure 4: Qualitative comparison of attention prediction. (Left) Visual Saliency models are distracted by high-contrast noise (the lamp). (Right) Our Semantic-Associative model correctly predicts fixation on the phone, driven by the semantic link from the user's hand, despite lower pixel saliency.
• Figure 5: Buffer stability during rapid narrative transitions. The baseline (red) suffers a stall event during a saccade at $t=4s$. The proposed method (blue) triggers an associative pre-fetch at $t=3.8s$, maintaining playback continuity.