Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical Vector-Quantized Latents for Perceptual Low-Resolution Video Compression

Manikanta Kotthapalli, Banafsheh Rekabdar

TL;DR

We propose MS‑VQ‑VAE, a two‑level hierarchical vector‑quantized autoencoder for perceptual, low‑resolution video compression, designed to generate compact latent codes for efficient storage and client‑side decoding. By extending VQ‑VAE‑2 to a spatiotemporal setting with 3D residuals and dual codebooks, and by incorporating a VGG‑16‑based perceptual loss, the method achieves improved rate–distortion and perceptual quality on short clips. On UCF101 with 2‑second segments, the approach attains PSNR 25.96 dB / SSIM 0.8375 and surpasses single‑scale baselines by about 1.4 dB PSNR and 0.024–0.03 SSIM, while delivering significant compression. The results demonstrate the practicality of hierarchical, perceptually guided latent compression for edge devices, mobile streaming, and CDN storage, with potential extensions to higher resolutions and explicit entropy modeling.

Abstract

The exponential growth of video traffic has placed increasing demands on bandwidth and storage infrastructure, particularly for content delivery networks (CDNs) and edge devices. While traditional video codecs like H.264 and HEVC achieve high compression ratios, they are designed primarily for pixel-domain reconstruction and lack native support for machine learning-centric latent representations, limiting their integration into deep learning pipelines. In this work, we present a Multi-Scale Vector Quantized Variational Autoencoder (MS-VQ-VAE) designed to generate compact, high-fidelity latent representations of low-resolution video, suitable for efficient storage, transmission, and client-side decoding. Our architecture extends the VQ-VAE-2 framework to a spatiotemporal setting, introducing a two-level hierarchical latent structure built with 3D residual convolutions. The model is lightweight (approximately 18.5M parameters) and optimized for 64x64 resolution video clips, making it appropriate for deployment on edge devices with constrained compute and memory resources. To improve perceptual reconstruction quality, we incorporate a perceptual loss derived from a pre-trained VGG16 network. Trained on the UCF101 dataset using 2-second video clips (32 frames at 16 FPS), on the test set we achieve 25.96 dB PSNR and 0.8375 SSIM. On validation, our model improves over the single-scale baseline by 1.41 dB PSNR and 0.0248 SSIM. The proposed framework is well-suited for scalable video compression in bandwidth-sensitive scenarios, including real-time streaming, mobile video analytics, and CDN-level storage optimization.

Hierarchical Vector-Quantized Latents for Perceptual Low-Resolution Video Compression

TL;DR

We propose MS‑VQ‑VAE, a two‑level hierarchical vector‑quantized autoencoder for perceptual, low‑resolution video compression, designed to generate compact latent codes for efficient storage and client‑side decoding. By extending VQ‑VAE‑2 to a spatiotemporal setting with 3D residuals and dual codebooks, and by incorporating a VGG‑16‑based perceptual loss, the method achieves improved rate–distortion and perceptual quality on short clips. On UCF101 with 2‑second segments, the approach attains PSNR 25.96 dB / SSIM 0.8375 and surpasses single‑scale baselines by about 1.4 dB PSNR and 0.024–0.03 SSIM, while delivering significant compression. The results demonstrate the practicality of hierarchical, perceptually guided latent compression for edge devices, mobile streaming, and CDN storage, with potential extensions to higher resolutions and explicit entropy modeling.

Abstract

The exponential growth of video traffic has placed increasing demands on bandwidth and storage infrastructure, particularly for content delivery networks (CDNs) and edge devices. While traditional video codecs like H.264 and HEVC achieve high compression ratios, they are designed primarily for pixel-domain reconstruction and lack native support for machine learning-centric latent representations, limiting their integration into deep learning pipelines. In this work, we present a Multi-Scale Vector Quantized Variational Autoencoder (MS-VQ-VAE) designed to generate compact, high-fidelity latent representations of low-resolution video, suitable for efficient storage, transmission, and client-side decoding. Our architecture extends the VQ-VAE-2 framework to a spatiotemporal setting, introducing a two-level hierarchical latent structure built with 3D residual convolutions. The model is lightweight (approximately 18.5M parameters) and optimized for 64x64 resolution video clips, making it appropriate for deployment on edge devices with constrained compute and memory resources. To improve perceptual reconstruction quality, we incorporate a perceptual loss derived from a pre-trained VGG16 network. Trained on the UCF101 dataset using 2-second video clips (32 frames at 16 FPS), on the test set we achieve 25.96 dB PSNR and 0.8375 SSIM. On validation, our model improves over the single-scale baseline by 1.41 dB PSNR and 0.0248 SSIM. The proposed framework is well-suited for scalable video compression in bandwidth-sensitive scenarios, including real-time streaming, mobile video analytics, and CDN-level storage optimization.
Paper Structure (33 sections, 1 equation, 2 figures, 3 tables)

This paper contains 33 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Video Compression
  4. Learned Image and Video Compression
  5. Vector Quantized Variational Autoencoders
  6. Perceptual Loss in Compression
  7. Multi-Scale and Hierarchical Models
  8. Methodology
  9. Multi-Scale Architecture
  10. Encoder.
  11. Quantization.
  12. Decoder.
  13. Residual Blocks and Temporal Modeling.
  14. Evaluation Metrics
  15. Perceptual Loss
  16. ...and 18 more sections

Figures (2)

  • Figure 1: MS-VQ-VAE architecture. The top encoder produces a coarse latent $\mathbf{z}_t$ that is refined by the bottom encoder into $\mathbf{z}_b$; both are quantized to $\hat{\mathbf{z}}_t,\hat{\mathbf{z}}_b$ and jointly decoded.
  • Figure 2: Qualitative comparison on UCF101 at $64\times64$. Columns: Original, VQ-VAE, VQ-VAE-2, and MS-VQ-VAE (ours).