PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation

TL;DR

PyraTok introduces Language-aligned Pyramidal Quantization (LaPQ) to address limitations of existing discrete video VAEs by hierarchically quantizing encoder features across multiple depths with a shared large binary codebook. It couples local, per-level text-conditioned alignment with a global autoregressive objective over the token sequence, ensuring strong cross-modal grounding and temporal coherence. Across ten benchmarks, PyraTok achieves state-of-the-art reconstruction and superior zero-shot performance in video segmentation, temporal action localization, and video understanding, while scaling effectively to 4K/8K resolutions. The approach provides a practical, general-purpose video tokenizer that improves both video generation quality and downstream video-language understanding, with robust transferability across pretrained VAEs and vision-language backbones.

Abstract

Discrete video VAEs underpin modern text-to-video generation and video understanding systems, yet existing tokenizers typically learn visual codebooks at a single scale with limited vocabularies and shallow language supervision, leading to poor cross-modal alignment and zero-shot transfer. We introduce PyraTok, a language-aligned pyramidal tokenizer that learns semantically structured discrete latents across multiple spatiotemporal resolutions. PyraTok builds on a pretrained video VAE and a novel Language aligned Pyramidal Quantization (LaPQ) module that discretizes encoder features at several depths using a shared large binary codebook, yielding compact yet expressive video token sequences. To tightly couple visual tokens with language, PyraTok jointly optimizes multi-scale text-guided quantization and a global autoregressive objective over the token hierarchy. Across ten benchmarks, PyraTok delivers state-of-the-art (SOTA) video reconstruction, consistently improves text-to-video quality, and sets new SOTA zero-shot performance on video segmentation, temporal action localization, and video understanding, scaling robustly to up to 4K/8K resolutions.

Figures (31)

• Figure 1: Given a video and text prompt, PyraTok encodes compact latents, facilitating high-quality reconstruction and a wide range of video-language understanding tasks.
• Figure 2: PyraTok attention maps illustrating fine-grained cross-modal alignment. Highlighted regions indicate language-guided semantic localization (e.g., Nike shoes, bikes).
• Figure 3: Overview of the proposed PyraTok architecture. Masked video frames are encoded and quantized at multiple scales via Language-aligned Pyramidal Quantization (LaPQ) blocks guided by text embeddings. The resulting multi-scale discrete tokens are aligned through a vision-language model for semantic consistency, enabling high-fidelity and text-aware video reconstruction.
• Figure 4: PCA projections of quantized tokens from each LaPQ's stage. Columns ($q^{(1)}$–$q^{(4)}$) show hierarchical outputs capturing progressively refined and semantically aligned regions.
• Figure 5: Frame reconstruction qualitative comparison. PyraTok generates sharper details, clearer textures, and better spatial structure than baselines, demonstrating better fidelity and semantic consistency.

Theorems & Definitions (2)

• Proposition 1: Non-optimality of Collapsed LaPQ Posteriors
• proof