Generative Human Video Compression with Multi-granularity Temporal Trajectory Factorization

TL;DR

This paper introduces Multi-granularity Temporal Trajectory Factorization (MTTF) to enable ultra-low bitrate generative human video compression. By factorizing motion into compact vectors and fine-grained motion fields, and coupling this with a resolution-expandable generator and foreground-background parallel generation, the framework delivers robust, scalable reconstruction across multiple resolutions. The approach achieves state-of-the-art rate-distortion performance against VVC and existing generative codecs on both moving-body and talking-face datasets, with strong subjective quality and resilience to resolution changes. The results indicate significant practical potential for bandwidth-constrained, high-quality human-centric video communication.

Abstract

In this paper, we propose a novel Multi-granularity Temporal Trajectory Factorization framework for generative human video compression, which holds great potential for bandwidth-constrained human-centric video communication. In particular, the proposed motion factorization strategy can facilitate to implicitly characterize the high-dimensional visual signal into compact motion vectors for representation compactness and further transform these vectors into a fine-grained field for motion expressibility. As such, the coded bit-stream can be entailed with enough visual motion information at the lowest representation cost. Meanwhile, a resolution-expandable generative module is developed with enhanced background stability, such that the proposed framework can be optimized towards higher reconstruction robustness and more flexible resolution adaptation. Experimental results show that proposed method outperforms latest generative models and the state-of-the-art video coding standard Versatile Video Coding (VVC) on both talking-face videos and moving-body videos in terms of both objective and subjective quality. The project page can be found at https://github.com/xyzysz/Extreme-Human-Video-Compression-with-MTTF.

Figures (10)

• Figure 1: Overview of proposed generative human video coding framework.
• Figure 2: The detailed diagram of multi-granularity temporal trajectory factorization.
• Figure 3: The detailed network structure of resolution-expandable generator. "$\uparrow$" denotes up-sample block, "$\downarrow$" denotes down-sample block and "$\rightarrow$" denotes blocks that maintain the feature size. "w" denotes warping with motion, "$\times$" denotes masking with occlusion. The network structure will be automatically initialized according to the depth and width setting. For example, the depth of network in this figure is set as 3. And if the width is 1, only modules with solid outline will be initialized. If the width is 3, all modules in the figure will be initialized.
• Figure 4: Overview of test set.
• Figure 5: Rate-distortion performance comparisons with VVC vvc, FOMM fomm, MRAA mraa, TPSM tpsm, CFTE cfte in terms of DISTS, LPIPS and FVD for moving-body test set.