STF: Spatio-Temporal Fusion Module for Improving Video Object Detection

TL;DR

The paper tackles the vulnerability of single-frame detectors in video contexts by exploiting temporal information across frames. It introduces three components—Multi-Frame Attention (MFA) for temporal/spatial weighting, Single-Frame Attention (SFA) for frame-wise refinement, and a Dual-Frame Fusion module for end-to-end feature integration—built on an HRNet backbone with a CenterNet-style head. The method is validated on Cityscapes, KITTI MOT, and UAVDT, with ablations showing substantial gains from each module and the proposed fusion strategy. The results demonstrate improved robustness to occlusion, motion blur, and scale variation, with code released for reproducibility.

Abstract

Consecutive frames in a video contain redundancy, but they may also contain relevant complementary information for the detection task. The objective of our work is to leverage this complementary information to improve detection. Therefore, we propose a spatio-temporal fusion framework (STF). We first introduce multi-frame and single-frame attention modules that allow a neural network to share feature maps between nearby frames to obtain more robust object representations. Second, we introduce a dual-frame fusion module that merges feature maps in a learnable manner to improve them. Our evaluation is conducted on three different benchmarks including video sequences of moving road users. The performed experiments demonstrate that the proposed spatio-temporal fusion module leads to improved detection performance compared to baseline object detectors. Code is available at https://github.com/noreenanwar/STF-module

Figures (3)

• Figure 1: Overview of our spatio-temporal based fusion framework (STF), illustrating the key components: MFA, SFA, and dual-fusion module
• Figure 2: Multi-Frame Attention module with multi-scale integrator.
• Figure 3: The channel and spatial attention modules of our proposed single-frame attention module