MaskVD: Region Masking for Efficient Video Object Detection

TL;DR

MaskVD tackles real-time video object detection by reducing redundant computation through region masking that leverages semantic cues and temporal consistency. It introduces a combined static and dynamic input mask and a backbone processing scheme that reuses features from prior frames, storing a single reference tensor per windowed block for ViTs while employing DeltaCNN-compatible kernels for CNNs. Empirically, it achieves up to $3.14\times$ FLOPs reduction and $1.5\times$ latency improvement on KITTI, with minimal $mAP\text{-}50$ degradation, and improves memory efficiency relative to the state of the art by up to about $2.3\times$. The approach is deployment-friendly, does not require re-training, and is compatible with both windowed and non-windowed ViT backbones as well as CNN backbones.

Abstract

Video tasks are compute-heavy and thus pose a challenge when deploying in real-time applications, particularly for tasks that require state-of-the-art Vision Transformers (ViTs). Several research efforts have tried to address this challenge by leveraging the fact that large portions of the video undergo very little change across frames, leading to redundant computations in frame-based video processing. In particular, some works leverage pixel or semantic differences across frames, however, this yields limited latency benefits with significantly increased memory overhead. This paper, in contrast, presents a strategy for masking regions in video frames that leverages the semantic information in images and the temporal correlation between frames to significantly reduce FLOPs and latency with little to no penalty in performance over baseline models. In particular, we demonstrate that by leveraging extracted features from previous frames, ViT backbones directly benefit from region masking, skipping up to 80% of input regions, improving FLOPs and latency by 3.14x and 1.5x. We improve memory and latency over the state-of-the-art (SOTA) by 2.3x and 1.14x, while maintaining similar detection performance. Additionally, our approach demonstrates promising results on convolutional neural networks (CNNs) and provides latency improvements over the SOTA up to 1.3x using specialized computational kernels.

Figures (9)

• Figure 1: mAP-50 vs. latency comparison of MaskVD with STGT liSpatiotemporalGated2021 and Eventful-Transformer Dutson_ICCV_2023, two SOTA token-dropping methods on ViTs for frame-based video object detection on ImageNet-VID dataset. MaskVD achieves the best mAP-50 latency trade-off.
• Figure 2: An example of generating static mask from ground truth bounding boxes in KITTI train set, where (a) shows the accumulated heatmap ($\mathcal{H}$), (b) shows region-wise scores, with heatmap values accumulated in regions of 16 $\times$16 pixels, and (c) shows our static mask with top 30% regions
• Figure 3: An example of generating combined mask from static and dynamic mask in KITTI. Union on (a) static mask with top 30% regions and (b) dynamic mask from bounding boxes of the last frame generated a (c) combined mask with top 35% regions.
• Figure 4: Inference framework of MaskVD with windowed ViT backbone. Input frame $t$ is fully processed, generating a set of reference tokens at the output. Input frame $t+1$ is masked following our proposed approach, feeding only a few tokens at the input, and the processed tokens updating the corresponding locations of the reference output. Figure \ref{['fig:window_attn']} shows W-MSA block blocks for masked frames.
• Figure 5: Windowed attention block (W-MSA) for processing masked frames. Each block maintains a set of reference tokens from the last processed frame, scattering received tokens at the input and gathering them at the output.