Content Adaptive Encoding For Interactive Game Streaming

TL;DR

This work tackles the challenge of content-adaptive encoding for interactive game streaming by introducing CAE-IGS, a lightweight CNN-based approach that predicts the next scene's resolution from past per-line HEVC statistics. The method operates without buffering or encoder lookahead, adapting only at scene changes to keep latency at a minimum. By exploring rate-quality convex hulls and training zone-specific predictors, CAE-IGS achieves statistically significant quality gains and bitrate reductions over a static HEVC ladder, approaching the offline Optimal Ladder while maintaining practical, millisecond-scale CPU inference. The results demonstrate CAE-IGS as a feasible, zero-latency CAE solution for IGS with real-world applicability and room for further extension.

Abstract

Video-on-demand streaming has benefitted from \textit{content-adaptive encoding} (CAE), i.e., adaptation of resolution and/or quantization parameters for each scene based on convex hull optimization. However, CAE is very challenging to develop and deploy for interactive game streaming (IGS). Commercial IGS services impose ultra-low latency encoding with no lookahead or buffering, and have extremely tight compute constraints for any CAE algorithm execution. We propose the first CAE approach for resolution adaptation in IGS based on compact encoding metadata from past frames. Specifically, we train a convolutional neural network (CNN) to infer the best resolution from the options available for the upcoming scene based on a running window of aggregated coding block statistics from the current scene. By deploying the trained CNN within a practical IGS setup based on HEVC encoding, our proposal: (i) improves over the default fixed-resolution ladder of HEVC by 2.3 Bjøntegaard Delta-VMAF points; (ii) infers using 1ms of a single CPU core per scene, thereby having no latency overhead.

Figures (5)

• Figure 1: Diagrammatic overview of the CAE-IGS framework.
• Figure 2: Convex hull composition in RQ space.
• Figure 3: Example of codec statistics: a 4K video frame is depicted along its SATD frame (averages of residual errors for each prediction block area) and the per-line stats. Every statistic comprises 34 values, as this is a 2160p frame that has 34 rows of HEVC CTBs.
• Figure 4: Proposed CNN model architecture. For each bitrate, the input comprises the 7 per-line stats (Fig. \ref{['fig:PerLineFeatures']}) of the previous $N=60$ frames (with each stat rescaled to 17 values). The output infers which of the two CAE-IGS resolutions of Table \ref{['tab: CronosLadder']} should be used depending on the bitrate zone.
• Figure 5: Examples of resulting ladders for four different test sequences that demonstrate most common improvement patterns of the proposed CAE-IGS and Optimal Ladder over the baseline Static Ladder solution.