Latent-INR: A Flexible Framework for Implicit Representations of Videos with Discriminative Semantics

TL;DR

This work introduces Latent-INR, a flexible video INR framework that decouples spatial and temporal components through a per-frame latent dictionary $\{z_t\}$ and video-specific hypernetworks that generate frame-specific INR weights. By low-rank modulating a shared base network, Latent-INR achieves competitive compression while enabling semantic tasks such as retrieval and interactive chat through alignment with CLIP and VideoLlama. The approach supports interpolation by latent-space blending and yields visually coherent frames at arbitrary resolutions (any-resolution inference). Empirical results on UVG and related datasets demonstrate competitive rate-distortion performance, strong interpolation fidelity, and compelling downstream capabilities, marking a first step toward semantically meaningful, open-structure video INRs that go beyond compression alone.

Abstract

Implicit Neural Networks (INRs) have emerged as powerful representations to encode all forms of data, including images, videos, audios, and scenes. With video, many INRs for video have been proposed for the compression task, and recent methods feature significant improvements with respect to encoding time, storage, and reconstruction quality. However, these encoded representations lack semantic meaning, so they cannot be used for any downstream tasks that require such properties, such as retrieval. This can act as a barrier for adoption of video INRs over traditional codecs as they do not offer any significant edge apart from compression. To alleviate this, we propose a flexible framework that decouples the spatial and temporal aspects of the video INR. We accomplish this with a dictionary of per-frame latents that are learned jointly with a set of video specific hypernetworks, such that given a latent, these hypernetworks can predict the INR weights to reconstruct the given frame. This framework not only retains the compression efficiency, but the learned latents can be aligned with features from large vision models, which grants them discriminative properties. We align these latents with CLIP and show good performance for both compression and video retrieval tasks. By aligning with VideoLlama, we are able to perform open-ended chat with our learned latents as the visual inputs. Additionally, the learned latents serve as a proxy for the underlying weights, allowing us perform tasks like video interpolation. These semantic properties and applications, existing simultaneously with ability to perform compression, interpolation, and superresolution properties, are a first in this field of work.

Figures (20)

• Figure 1: Existing INRs for video (left) typically take some time-coordinate, or time and positional coordinates and train a single network to reconstruct a video. In contrast to these, we propose an INR system where a dictionary of implicit latent codes is learned for a video, one latent per frame. The latents are aligned to the image features of a large vision model, while simultaneously an INR system is learned which, given these latent codes, generates a positional INR which can reconstruct the frame. With this framework, we successfully develop an INR which performs both reconstructive tasks like compression, and semantic downstream tasks like retrieval and interactive chat.
• Figure 2: We propose a new framework for video INR models by decoupling the spatial and temporal aspects of modeling. Our framework consists of auto-decoder based learnable latents that modulate the base network using a hypernetwork, via low-rank modulation. Once encoded, the resulting latents act as a proxy for the underlying weights of the representation. On the right, we show the use of these latents for additional tasks like video interpolation. By aligning these latents to the embedding space of foundational models like CLIP and VideoLlama, we also perform retrieval and chat.
• Figure 3: We plot the rate distortion curves on PSNR and SSIM to compare compression with other methods. We observe that our large model achieves comparable PSNR to the current SOTA kim2022scalable. Note that, while not plotted here, our decoding FPS is superior. Additional per-video results are available in the Supplementary.
• Figure 4: With the same model, we can perform inference at any resolution, with speeds competitive or beating HEVC. We show sample frames for each resolution.
• Figure 5: We achieve high quality reconstruction and are able to reproduce even the finer details like water fountains and the hair on the horse.