Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Flow-Aware Diffusion for Real-Time VR Restoration: Enhancing Spatiotemporal Coherence and Efficiency

Yitong Zhu, Qianghong Dong, Guanxuan Jiang, Zhuowen Liang, Yuyang Wang

TL;DR

This work tackles cybersickness in VR by addressing mismatches in visual motion, particularly optical flow, which disrupts vestibular-visual coherence. It introduces U-MAD, a flow-guided diffusion-based video restoration framework that operates at the image level with a plug-and-play U-shaped Mamba backbone and a Unified Motion-Structure Embedding (UMSE) to condition denoising on motion and structural priors. The system combines a Global Context Module, Post-Temporal Context Module, and flow-aware conditioning within a diffusion objective that jointly optimizes reconstruction, temporal coherence, and flow consistency, achieving real-time performance. Empirical results on VR and omnidirectional datasets show improved temporal stability and significant reductions in cybersickness in a user study, indicating practical impact for safer, more comfortable immersive experiences.

Abstract

Cybersickness remains a critical barrier to the widespread adoption of Virtual Reality (VR), particularly in scenarios involving intense or artificial motion cues. Among the key contributors is excessive optical flow-perceived visual motion that, when unmatched by vestibular input, leads to sensory conflict and discomfort. While previous efforts have explored geometric or hardware based mitigation strategies, such methods often rely on predefined scene structures, manual tuning, or intrusive equipment. In this work, we propose U-MAD, a lightweight, real-time, AI-based solution that suppresses perceptually disruptive optical flow directly at the image level. Unlike prior handcrafted approaches, this method learns to attenuate high-intensity motion patterns from rendered frames without requiring mesh-level editing or scene specific adaptation. Designed as a plug and play module, U-MAD integrates seamlessly into existing VR pipelines and generalizes well to procedurally generated environments. The experiments show that U-MAD consistently reduces average optical flow and enhances temporal stability across diverse scenes. A user study further confirms that reducing visual motion leads to improved perceptual comfort and alleviated cybersickness symptoms. These findings demonstrate that perceptually guided modulation of optical flow provides an effective and scalable approach to creating more user-friendly immersive experiences. The code will be released at https://github.com/XXXXX (upon publication).

Flow-Aware Diffusion for Real-Time VR Restoration: Enhancing Spatiotemporal Coherence and Efficiency

TL;DR

This work tackles cybersickness in VR by addressing mismatches in visual motion, particularly optical flow, which disrupts vestibular-visual coherence. It introduces U-MAD, a flow-guided diffusion-based video restoration framework that operates at the image level with a plug-and-play U-shaped Mamba backbone and a Unified Motion-Structure Embedding (UMSE) to condition denoising on motion and structural priors. The system combines a Global Context Module, Post-Temporal Context Module, and flow-aware conditioning within a diffusion objective that jointly optimizes reconstruction, temporal coherence, and flow consistency, achieving real-time performance. Empirical results on VR and omnidirectional datasets show improved temporal stability and significant reductions in cybersickness in a user study, indicating practical impact for safer, more comfortable immersive experiences.

Abstract

Cybersickness remains a critical barrier to the widespread adoption of Virtual Reality (VR), particularly in scenarios involving intense or artificial motion cues. Among the key contributors is excessive optical flow-perceived visual motion that, when unmatched by vestibular input, leads to sensory conflict and discomfort. While previous efforts have explored geometric or hardware based mitigation strategies, such methods often rely on predefined scene structures, manual tuning, or intrusive equipment. In this work, we propose U-MAD, a lightweight, real-time, AI-based solution that suppresses perceptually disruptive optical flow directly at the image level. Unlike prior handcrafted approaches, this method learns to attenuate high-intensity motion patterns from rendered frames without requiring mesh-level editing or scene specific adaptation. Designed as a plug and play module, U-MAD integrates seamlessly into existing VR pipelines and generalizes well to procedurally generated environments. The experiments show that U-MAD consistently reduces average optical flow and enhances temporal stability across diverse scenes. A user study further confirms that reducing visual motion leads to improved perceptual comfort and alleviated cybersickness symptoms. These findings demonstrate that perceptually guided modulation of optical flow provides an effective and scalable approach to creating more user-friendly immersive experiences. The code will be released at https://github.com/XXXXX (upon publication).

Paper Structure

This paper contains 33 sections, 9 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cybersickness and Visual Motion: A Sensory Conflict Perspective
  4. Cybersickness and optical flow
  5. Explicit Geometry-Based Flow Reduction
  6. Learning-Based Flow-Aware Modeling
  7. Methodology
  8. Global and Temporal Context Modules
  9. U-shaped Mamba Diffusion Backbone
  10. Unified Motion-Structure Embedding (UMSE)
  11. Flow-guided Motion Encoding.
  12. Location-aware Structural Encoding.
  13. Conditional Embedding Injection.
  14. Training Objective
  15. Experiments
  16. ...and 18 more sections

Figures (6)

  • Figure 1: Framework of the proposed U-MAD
  • Figure 2: The detailed architecture of the TSC block.
  • Figure 3: Overview of the UMSE module
  • Figure 4: Performance over training epochs for each backbone. U-MAD exhibits more stable convergence and higher final accuracy across all metrics.
  • Figure 5: Visual comparison of different conditioning strategies: (a) Original degraded input; (b) Motion-based condition (MotionCtrl); (c) Camera parameter condition (CameraCtrl); (d) Our flow-guided condition. Each row shows restored frames, optical flow, temporal consistency, and consistency error maps. Our method achieves superior temporal alignment and smoother motion trajectories.
  • ...and 1 more figures