Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SCENE: Semantic-aware Codec Enhancement with Neural Embeddings

Han-Yu Lin, Li-Wei Chen, Hung-Shin Lee

TL;DR

SCENE addresses perceptual degradation from standard video codecs by introducing a semantic-aware pre-processing stage that leverages vision-language embeddings to guide assembled convolutions. The method is trained end-to-end with a differentiable codec proxy, enabling codec-robust enhancement without altering the encoding pipeline, and operates as a standalone pre-processor during inference for real-time use. Key contributions include a per-channel, semantic-conditioned assembled convolution mechanism, a SigLIP 2–based semantic feature extractor, and a differentiable JPEG-based training objective that jointly optimizes perceptual quality and bitrate considerations, achieving notable VMAF gains at reduced bitrates on H.264 and H.265 baselines. Empirically, SCENE provides consistent perceptual improvements (VMAF) with comparable MS-SSIM, while maintaining real-time performance (~36 fps at 1080p on consumer-grade GPUs), demonstrating the practicality of semantic-guided codec enhancement as a pre-processing step in streaming pipelines.

Abstract

Compression artifacts from standard video codecs often degrade perceptual quality. We propose a lightweight, semantic-aware pre-processing framework that enhances perceptual fidelity by selectively addressing these distortions. Our method integrates semantic embeddings from a vision-language model into an efficient convolutional architecture, prioritizing the preservation of perceptually significant structures. The model is trained end-to-end with a differentiable codec proxy, enabling it to mitigate artifacts from various standard codecs without modifying the existing video pipeline. During inference, the codec proxy is discarded, and SCENE operates as a standalone pre-processor, enabling real-time performance. Experiments on high-resolution benchmarks show improved performance over baselines in both objective (MS-SSIM) and perceptual (VMAF) metrics, with notable gains in preserving detailed textures within salient regions. Our results show that semantic-guided, codec-aware pre-processing is an effective approach for enhancing compressed video streams.

SCENE: Semantic-aware Codec Enhancement with Neural Embeddings

TL;DR

SCENE addresses perceptual degradation from standard video codecs by introducing a semantic-aware pre-processing stage that leverages vision-language embeddings to guide assembled convolutions. The method is trained end-to-end with a differentiable codec proxy, enabling codec-robust enhancement without altering the encoding pipeline, and operates as a standalone pre-processor during inference for real-time use. Key contributions include a per-channel, semantic-conditioned assembled convolution mechanism, a SigLIP 2–based semantic feature extractor, and a differentiable JPEG-based training objective that jointly optimizes perceptual quality and bitrate considerations, achieving notable VMAF gains at reduced bitrates on H.264 and H.265 baselines. Empirically, SCENE provides consistent perceptual improvements (VMAF) with comparable MS-SSIM, while maintaining real-time performance (~36 fps at 1080p on consumer-grade GPUs), demonstrating the practicality of semantic-guided codec enhancement as a pre-processing step in streaming pipelines.

Abstract

Compression artifacts from standard video codecs often degrade perceptual quality. We propose a lightweight, semantic-aware pre-processing framework that enhances perceptual fidelity by selectively addressing these distortions. Our method integrates semantic embeddings from a vision-language model into an efficient convolutional architecture, prioritizing the preservation of perceptually significant structures. The model is trained end-to-end with a differentiable codec proxy, enabling it to mitigate artifacts from various standard codecs without modifying the existing video pipeline. During inference, the codec proxy is discarded, and SCENE operates as a standalone pre-processor, enabling real-time performance. Experiments on high-resolution benchmarks show improved performance over baselines in both objective (MS-SSIM) and perceptual (VMAF) metrics, with notable gains in preserving detailed textures within salient regions. Our results show that semantic-guided, codec-aware pre-processing is an effective approach for enhancing compressed video streams.
Paper Structure (12 sections, 3 equations, 3 figures, 1 table)

This paper contains 12 sections, 3 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Semantic Feature Extraction
  4. Assembled Convolutions
  5. Training Strategy
  6. Experiments
  7. Dataset and Preprocessing
  8. Implementation Details
  9. Evaluation Protocol
  10. Baseline Methods
  11. Results and Analysis
  12. Conclusion and Future Work

Figures (3)

  • Figure 1: Illustration of the proposed SCENE framework.
  • Figure 2: Qualitative comparison between standard AV1 and our SCENE-enhanced AV1.
  • Figure 3: Rate-distortion curves for H.264, H.265, and AV1 on MS-SSIM, VMAF_ NEG, and VMAF, respectively.