Efficient Neural Video Representation with Temporally Coherent Modulation

TL;DR

This work tackles the inefficiency of existing implicit neural video representations (INR) by introducing NVTM, which captures video dynamics with temporally coherent modulation. The method segments videos into GOP units, learns a time-aware alignment flow, and uses per-GOP 2D latent grids to modulate a base decoder, achieving fast encoding and improved parameter efficiency. NVTM demonstrates strong performance across video reconstruction, super-resolution, frame interpolation, inpainting, and compression, surpassing NeRV-style methods in speed and surpassing grid-based approaches in quality with fewer parameters. The results suggest practical impact for real-time INR-based video processing and compression, with robust ablations and analyses supporting the design choices.

Abstract

Implicit neural representations (INR) has found successful applications across diverse domains. To employ INR in real-life, it is important to speed up training. In the field of INR for video applications, the state-of-the-art approach employs grid-type parametric encoding and successfully achieves a faster encoding speed in comparison to its predecessors. However, the grid usage, which does not consider the video's dynamic nature, leads to redundant use of trainable parameters. As a result, it has significantly lower parameter efficiency and higher bitrate compared to NeRV-style methods that do not use a parametric encoding. To address the problem, we propose Neural Video representation with Temporally coherent Modulation (NVTM), a novel framework that can capture dynamic characteristics of video. By decomposing the spatio-temporal 3D video data into a set of 2D grids with flow information, NVTM enables learning video representation rapidly and uses parameter efficiently. Our framework enables to process temporally corresponding pixels at once, resulting in the fastest encoding speed for a reasonable video quality, especially when compared to the NeRV-style method, with a speed increase of over 3 times. Also, it remarks an average of 1.54dB/0.019 improvements in PSNR/LPIPS on UVG (Dynamic) (even with 10% fewer parameters) and an average of 1.84dB/0.013 improvements in PSNR/LPIPS on MCL-JCV (Dynamic), compared to previous grid-type works. By expanding this to compression tasks, we demonstrate comparable performance to video compression standards (H.264, HEVC) and recent INR approaches for video compression. Additionally, we perform extensive experiments demonstrating the superior performance of our algorithm across diverse tasks, encompassing super resolution, frame interpolation and video inpainting. Project page is https://sujiikim.github.io/NVTM/.

Figures (13)

• Figure 1: Fast encoding speed with high image quality. (Left) The encoding speed in UVG, where all models are configured at 0.1bpp and evaluated on the same resource conditions. NVTM learns quickly and achieves 30dB 3$\times$faster than the NeRV-series. (Right) Video reconstruction results on ReadySetGo sequence after training for 1 minutes. While E-NeRV and HNeRV exhibit blurry outputs, NVP and NVTM, based on parametric encoding, quickly capture complex representations. Further, NVTM excels at representing fine details such as text and numbers.
• Figure 2: Overview of NVTM. NVTM generates the same modulation latent for temporally correlated pixels between consecutive frames, and the latent is used to modulate the base network. To obtain this latent, 1) input video is split into GOP units, 2) network $F$ generates an alignment flow to transform 3D coordinate $(x,y,t)$ to specific time $t_k$ in $k$-th GOP unit, 3) 2D aligned coordinated $(x_{k}, y_{k})$ is obtained by adding $(x,y)$ and the alignment flow. 4) The temporally coherent latent $(z_{xyt})$ is extracted from the latent grid $G_{k}$ using normalized $(x'_{k}, y'_{k})$. Following the process, the temporally correlated 3D coordinates (yellow square and orange square) are mapped to the same 2D coordinate, thereby ensuring they share the same modulation latent representation. This shared modulation helps in the fast and efficient learning of video representation.
• Figure 3: Video inpainting and compression performance. (a) Visualization of video inpainting on Blackswan and Camel sequences in DAVIS2017. Although the masked regions are excluded during encoding, the NVTM successfully decodes them by utilizing temporally coherent modulation latent from adjacent frames. (b) BPP-PSNR plot of video compression on UVG (Dynamic). We encode all models with each video sequence and evaluate as following authors guided.
• Figure 4: (a) t-SNE visualization of modulation latent $z_{xyz}$ from our alignment module on corresponding pixels (1${st}$ and 5${th}$ frame). We select areas with similar pixel information, i.e. RGB values, and for ease of verification, these are denoted as {Horse, Grass, Sign}. The latent derived from the 1${st}$ frame and 5$th$ are marked with circle and star respectively. The analysis is based on segments, each consisting of 400 pixels. (b) Effects of alignment flow. Each line represents the performance with replacing our alignment method on Bosphorus sequence. Purple indicates aligning with zero-valued flow (i.e., its spatial coordinate). Green and blue indicates aligning with random-valued flow in a notated scale of source video resolution.
• Figure 5: Visualization results of video inpainting. Each are the first frame of Camel, Blackswan, Cows, Drift-Chicane, Soapbox and Tennis sequence on DAVIS2017.