PNeRV: A Polynomial Neural Representation for Videos

TL;DR

Polynomial Neural Representation for Videos (PNeRV), a parameter-wise efficient, patch-wise INR for videos that preserves spatiotemporal continuity, and a custom Hierarchical Patch-wise Spatial Sampling Scheme that ensures spatial continuity while retaining parameter efficiency.

Abstract

Extracting Implicit Neural Representations (INRs) on video data poses unique challenges due to the additional temporal dimension. In the context of videos, INRs have predominantly relied on a frame-only parameterization, which sacrifices the spatiotemporal continuity observed in pixel-level (spatial) representations. To mitigate this, we introduce Polynomial Neural Representation for Videos (PNeRV), a parameter-wise efficient, patch-wise INR for videos that preserves spatiotemporal continuity. PNeRV leverages the modeling capabilities of Polynomial Neural Networks to perform the modulation of a continuous spatial (patch) signal with a continuous time (frame) signal. We further propose a custom Hierarchical Patch-wise Spatial Sampling Scheme that ensures spatial continuity while retaining parameter efficiency. We also employ a carefully designed Positional Embedding methodology to further enhance PNeRV's performance. Our extensive experimentation demonstrates that PNeRV outperforms the baselines in conventional Implicit Neural Representation tasks like compression along with downstream applications that require spatiotemporal continuity in the underlying representation. PNeRV not only addresses the challenges posed by video data in the realm of INRs but also opens new avenues for advanced video processing and analysis.

Figures (17)

• Figure 1: PNeRV when compared to its counterparts: (a) NeRV: An INR for videos with only frame-wise parameterization that leads to loss of spatial continuity. (b) E-NeRV: A step-up over NeRV with a parameterization that employs a fixed Spatial Context (SC). The fixed SC does not support spatial continuity. (c) PNeRV: An efficient INR for videos with a PNN backbone (signified by the usage of Hadamard Product $\odot$) that supports varying SC while retaining spatial continuity.
• Figure 2: The PNeRV Architecture: The PNeRV pipeline consists of three modules. First, the PEs of time index $t$, coarse patch coordinate $\boldsymbol{\lambda_{ij}}$ and the fine patch coordinate $\mathbf{\Lambda_{ij}}$ are computed in the Positional Embedding Module. Second, these embeddings are fused effectively in the Embedding Fusion Block. Finally the PNN-based INR decoder reconstructs the frame patch, given a fused Positional Embedding $z$. Here FC denotes a fully connected layer of appropriate input-output dimensions.
• Figure 3: Hierarchical Patch-wise Spatial Sampling: (a) A Global coordinate grid $\mathcal{C}$ with input values normalized to range $[0,1]$ is constructed for each frame. (b) The grid is divided into $M \times N$ coarse patches of equal size. For a coarse patch $\boldsymbol{P_{ij}}$, its centroid is used as a 2D coordinate $\boldsymbol{\lambda_{ij}}$. (c) Each coarse patch is further divided into $K \times L$ fine patches and a collection of the centroids of these smaller patches is used as the fine patch coordinate tensor $\boldsymbol{\Lambda_{ij}}$.
• Figure 4: The HMF architecture at a glance: All linear transformation matrices represent the terms in Eq. \ref{['eq: HMF']}. Here, $\odot$ denotes the Hadamard Product, $\bigoplus$ represents feature addition, black arrows represent inputs, and blue arrows represent the fused entities.
• Figure 5: Model pruning results on NeRV-L, E-NeRV and PNeRV trained for 300 epochs on "Big Buck Bunny" video. Sparsity represents the ratio of pruned parameters.