Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeRV360: Neural Representation for 360-Degree Videos with a Viewport Decoder

Daichi Arai, Kyohei Unno, Yasuko Sugito, Yuichi Kusakabe

TL;DR

NeRV360 tackles the prohibitive memory and decoding costs of applying implicit neural video representations to 360° content by enabling direct viewport decoding. It integrates viewport extraction into the decoding step, leveraging a ConvNeXt encoder, perspective-projected viewport extraction, and a viewpoint-conditioned STAT module (with a channel-expansion strategy to mitigate embedding interpolation) to reconstruct only the user-selected region. Key contributions include a viewport decoder, a channel-expansion layer before extraction, and a viewpoint-conditioned spatial–temporal affine transform, validated on $6K$-resolution sequences where NeRV360 achieves $7×$ memory savings and $2.5×$ faster decoding with superior image quality. These results indicate substantial practical benefits for real-time VR/video delivery on GPUs with limited memory, and the approach scales toward ultra-high-resolution content such as $8K$ and beyond.

Abstract

Implicit neural representations for videos (NeRV) have shown strong potential for video compression. However, applying NeRV to high-resolution 360-degree videos causes high memory usage and slow decoding, making real-time applications impractical. We propose NeRV360, an end-to-end framework that decodes only the user-selected viewport instead of reconstructing the entire panoramic frame. Unlike conventional pipelines, NeRV360 integrates viewport extraction into decoding and introduces a spatial-temporal affine transform module for conditional decoding based on viewpoint and time. Experiments on 6K-resolution videos show that NeRV360 achieves a 7-fold reduction in memory consumption and a 2.5-fold increase in decoding speed compared to HNeRV, a representative prior work, while delivering better image quality in terms of objective metrics.

NeRV360: Neural Representation for 360-Degree Videos with a Viewport Decoder

TL;DR

NeRV360 tackles the prohibitive memory and decoding costs of applying implicit neural video representations to 360° content by enabling direct viewport decoding. It integrates viewport extraction into the decoding step, leveraging a ConvNeXt encoder, perspective-projected viewport extraction, and a viewpoint-conditioned STAT module (with a channel-expansion strategy to mitigate embedding interpolation) to reconstruct only the user-selected region. Key contributions include a viewport decoder, a channel-expansion layer before extraction, and a viewpoint-conditioned spatial–temporal affine transform, validated on -resolution sequences where NeRV360 achieves memory savings and faster decoding with superior image quality. These results indicate substantial practical benefits for real-time VR/video delivery on GPUs with limited memory, and the approach scales toward ultra-high-resolution content such as and beyond.

Abstract

Implicit neural representations for videos (NeRV) have shown strong potential for video compression. However, applying NeRV to high-resolution 360-degree videos causes high memory usage and slow decoding, making real-time applications impractical. We propose NeRV360, an end-to-end framework that decodes only the user-selected viewport instead of reconstructing the entire panoramic frame. Unlike conventional pipelines, NeRV360 integrates viewport extraction into decoding and introduces a spatial-temporal affine transform module for conditional decoding based on viewpoint and time. Experiments on 6K-resolution videos show that NeRV360 achieves a 7-fold reduction in memory consumption and a 2.5-fold increase in decoding speed compared to HNeRV, a representative prior work, while delivering better image quality in terms of objective metrics.
Paper Structure (10 sections, 2 equations, 4 figures, 2 tables)

This paper contains 10 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Viewport Extraction
  5. Viewport Decoder
  6. Experiments
  7. Experimental Setup
  8. Experimental Results
  9. Conclusion
  10. References

Figures (4)

  • Figure 1: Comparison of pipelines: (a) conventional decoding followed by viewport extraction matoba2019vr8k, and (b) NeRV360 decoding with integrated viewport extraction.
  • Figure 2: Overview of the NeRV360 framework. The input 360-degree frame $x_t$ is represented as $(3, H, W)$ and the output viewport $\hat{x}^{\mathrm{vp}}_{t,\theta,\varphi}$ as $(3, H^{\mathrm{vp}}, W^{\mathrm{vp}})$. The encoder and decoder strides are $(3, 2, 2, 2)$.
  • Figure 3: Illustration of our viewport decoder and STAT.
  • Figure 4: Comparison of NeRV360, HNeRV 10205438, and HNeRV-Boost 10658449 for video regression with equivalent model sizes.