Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HybridPrompt: Bridging Generative Priors and Traditional Codecs for Mobile Streaming

Liming Liu, Jiangkai Wu, Haoyang Wang, Peiheng Wang, Zongming Guo, Xinggong Zhang

TL;DR

This work presents HybridPrompt, the first generative-based video system capable of achieving real-time 1080p decoding at over 150 FPS on a commercial smartphone, using a hybrid architecture that encodes Keyframes using a generative model while relying on traditional codecs for the remaining frames.

Abstract

In Video on Demand (VoD) scenarios, traditional codecs are the industry standard due to their high decoding efficiency. However, they suffer from severe quality degradation under low bandwidth conditions. While emerging generative neural codecs offer significantly higher perceptual quality, their reliance on heavy frame-by-frame generation makes real-time playback on mobile devices impractical. We ask: is it possible to combine the blazing-fast speed of traditional standards with the superior visual fidelity of neural approaches? We present HybridPrompt, the first generative-based video system capable of achieving real-time 1080p decoding at over 150 FPS on a commercial smartphone. Specifically, we employ a hybrid architecture that encodes Keyframes using a generative model while relying on traditional codecs for the remaining frames. A major challenge is that the two paradigms have conflicting objectives: the "hallucinated" details from generative models often misalign with the rigid prediction mechanisms of traditional codecs, causing bitrate inefficiency. To address this, we demonstrate that the traditional decoding process is differentiable, enabling an end-to-end optimization loop. This allows us to use subsequent frames as additional supervision, forcing the generative model to synthesize keyframes that are not only perceptually high-fidelity but also mathematically optimal references for the traditional codec. By integrating a two-stage generation strategy, our system outperforms pure neural baselines by orders of magnitude in speed while achieving an average LPIPS gain of 8% over traditional codecs at 200kbps.

HybridPrompt: Bridging Generative Priors and Traditional Codecs for Mobile Streaming

TL;DR

This work presents HybridPrompt, the first generative-based video system capable of achieving real-time 1080p decoding at over 150 FPS on a commercial smartphone, using a hybrid architecture that encodes Keyframes using a generative model while relying on traditional codecs for the remaining frames.

Abstract

In Video on Demand (VoD) scenarios, traditional codecs are the industry standard due to their high decoding efficiency. However, they suffer from severe quality degradation under low bandwidth conditions. While emerging generative neural codecs offer significantly higher perceptual quality, their reliance on heavy frame-by-frame generation makes real-time playback on mobile devices impractical. We ask: is it possible to combine the blazing-fast speed of traditional standards with the superior visual fidelity of neural approaches? We present HybridPrompt, the first generative-based video system capable of achieving real-time 1080p decoding at over 150 FPS on a commercial smartphone. Specifically, we employ a hybrid architecture that encodes Keyframes using a generative model while relying on traditional codecs for the remaining frames. A major challenge is that the two paradigms have conflicting objectives: the "hallucinated" details from generative models often misalign with the rigid prediction mechanisms of traditional codecs, causing bitrate inefficiency. To address this, we demonstrate that the traditional decoding process is differentiable, enabling an end-to-end optimization loop. This allows us to use subsequent frames as additional supervision, forcing the generative model to synthesize keyframes that are not only perceptually high-fidelity but also mathematically optimal references for the traditional codec. By integrating a two-stage generation strategy, our system outperforms pure neural baselines by orders of magnitude in speed while achieving an average LPIPS gain of 8% over traditional codecs at 200kbps.
Paper Structure (16 sections, 7 figures, 4 tables)

This paper contains 16 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivation and Key Insights
  3. The Dilemma: Quality vs. Efficiency in Mobile VoD
  4. Feasibility of Hybrid Architecture
  5. The Challenge: The Perception-Precision Misalignment
  6. Insight: Bridging the Gap via Differentiable End-to-End Optimization
  7. Speed Optimization: Decoupling Resolution for Real-Time Decoding
  8. System Design
  9. Architecture Overview
  10. Server-Side: Transcoding Pipeline
  11. Client-Side: Parallel Decoding
  12. Evaluation
  13. Experimental Setup
  14. Perceptual Quality Analysis
  15. System Efficiency & Ablation Study
  16. ...and 1 more sections

Figures (7)

  • Figure 1: Overview of HybridPrompt.
  • Figure 2: We compressed (H.265) a 1080p@30fps video with a group of picture (GOP) length of 120 frames at different bitrates. It can be observed that at lower bitrates, block artifacts and blurring become more pronounced, resulting in noticeably degraded quality.
  • Figure 3: The Perception-Precision Misalignment. All three I-frames are compressed to an identical size of 8.8 KB. Each subfigure displays the RGB frame (Top) and its corresponding MSE Heatmap (Bottom).
  • Figure 4: Runtime analysis of diffusion-based generation under different resolutions, measured on an iPhone 16 Pro Max. Lower resolutions correspond to significantly reduced latency.
  • Figure 5: Perceptual quality (LPIPS, lower is better) comparison across different bitrate settings at 1080p resolution. HybridPrompt consistently achieves the lowest LPIPS (best quality) under all bandwidth conditions.
  • ...and 2 more figures