See Without Decoding: Motion-Vector-Based Tracking in Compressed Video

TL;DR

The paper tackles the challenge of real-time surveillance analytics by eliminating the need to decode full RGB frames. It introduces a hybrid compressed-domain tracking pipeline that decodes only the I-frame for an initial detection and propagates bounding boxes across subsequent P-frames using codec-domain cues, via a Box-Aligned Feature Extraction (BAFE) module and a BiLSTM-based temporal head. Empirically, the MV+DCT configuration achieves 0.8962 mAP on MOT17 (within ~2.25% of RGB-based baselines and +2.86% over Mean MV) and delivers substantial throughput gains (up to 3.7x) with a compact parameter count (~160K), enabling many more simultaneous streams. This work demonstrates the practicality of codec-domain motion modeling for scalable, energy-efficient real-time analytics in large camera networks, with potential extensions to multi-scale grids and larger GOPs to further enhance robustness.

Abstract

We propose a lightweight compressed-domain tracking model that operates directly on video streams, without requiring full RGB video decoding. Using motion vectors and transform coefficients from compressed data, our deep model propagates object bounding boxes across frames, achieving a computational speed-up of order up to 3.7 with only a slight 4% mAP@0.5 drop vs RGB baseline on MOTS15/17/20 datasets. These results highlight codec-domain motion modeling efficiency for real-time analytics in large monitoring systems.

Figures (1)

• Figure 1: BAFE architecture: box-aligned MV/DCT features are fused with box encodings and then fed to BiLSTM for temporal modeling.