Morphe: High-Fidelity Generative Video Streaming with Vision Foundation Model

TL;DR

Morphe tackles the challenge of delivering high-fidelity, real-time video over bandwidth-constrained networks by leveraging Vision Foundation Models in an end-to-end streaming system. The approach fuses a Visual-enhanced Generative Codec with a Resolution Scaling Accelerator and a Network-Adaptive Streaming Controller to achieve substantial bitrate reductions (≈62.5% vs H.265) while maintaining perceptual quality and real-time performance. Key contributions include asymmetric spatiotemporal compression, temporal smoothing, scalable token-based coding, and a network-aware token-dropping and recovery strategy that preserves core semantics under loss. The results show strong visual quality, robustness to packet loss, and practical edge-device deployment, illustrating the viability of VFM-powered streaming for real-world multimedia delivery.

Abstract

Video streaming is a fundamental Internet service, while the quality still cannot be guaranteed especially in poor network conditions such as bandwidth-constrained and remote areas. Existing works mainly work towards two directions: traditional pixel-codec streaming nearly approaches its limit and is hard to step further in compression; the emerging neural-enhanced or generative streaming usually fall short in latency and visual fidelity, hindering their practical deployment. Inspired by the recent success of vision foundation model (VFM), we strive to harness the powerful video understanding and processing capacities of VFM to achieve generalization, high fidelity and loss resilience for real-time video streaming with even higher compression rate. We present the first revolutionized paradigm that enables VFM-based end-to-end generative video streaming towards this goal. Specifically, Morphe employs joint training of visual tokenizers and variable-resolution spatiotemporal optimization under simulated network constraints. Additionally, a robust streaming system is constructed that leverages intelligent packet dropping to resist real-world network perturbations. Extensive evaluation demonstrates that Morphe achieves comparable visual quality while saving 62.5\% bandwidth compared to H.265, and accomplishes real-time, loss-resilient video delivery in challenging network environments, representing a milestone in VFM-enabled multimedia streaming solutions.

Figures (17)

• Figure 1: Case Study of Bandwidth-Constrained Multimedia Communication.
• Figure 2: Visual perception of different streaming technologies at 400 kbps . Most existing solutions show severe artifacts under poor network conditions.
• Figure 3: Video tokenization pipeline in Vision Foundation Models. An input video is encoded into tokens, which are usually much more compact than the input video. The decoder then reconstructs the input video from the tokens.
• Figure 4: Morphe Overview. It consists of three core modules: Visual-enhanced Generative Codec (§\ref{['model design']}), Resolution Scaling Accelerator (§\ref{['downsample']}), and Network-Adaptive Streaming Controller (§\ref{['network']}).
• Figure 5: Illustration of Similarity-based Token Selection.