TGBFormer: Transformer-GraphFormer Blender Network for Video Object Detection

TL;DR

This paper tackles video object detection by unifying global temporal context and local spatial-temporal cues. It introduces TGBFormer, which integrates a Spatial-Temporal Transformer for global representations, a Spatial-Temporal GraphFormer for local representations, and a Global-Local Feature Blender to adaptively fuse them in a parallel sequence-wise detection framework. Through extensive experiments on ImageNet VID, it achieves state-of-the-art results (e.g., 86.5% mAP with ResNet-101) while delivering real-time performance, demonstrating the benefit of combining long-range and short-range temporal information. The approach offers a practical impact by providing robust detection under motion blur and occlusion, thanks to its complementary global and local feature modeling.

Abstract

Video object detection has made significant progress in recent years thanks to convolutional neural networks (CNNs) and vision transformers (ViTs). Typically, CNNs excel at capturing local features but struggle to model global representations. Conversely, ViTs are adept at capturing long-range global features but face challenges in representing local feature details. Off-the-shelf video object detection methods solely rely on CNNs or ViTs to conduct feature aggregation, which hampers their capability to simultaneously leverage global and local information, thereby resulting in limited detection performance. In this paper, we propose a Transformer-GraphFormer Blender Network (TGBFormer) for video object detection, with three key technical improvements to fully exploit the advantages of transformers and graph convolutional networks while compensating for their limitations. First, we develop a spatial-temporal transformer module to aggregate global contextual information, constituting global representations with long-range feature dependencies. Second, we introduce a spatial-temporal GraphFormer module that utilizes local spatial and temporal relationships to aggregate features, generating new local representations that are complementary to the transformer outputs. Third, we design a global-local feature blender module to adaptively couple transformer-based global representations and GraphFormer-based local representations. Extensive experiments demonstrate that our TGBFormer establishes new state-of-the-art results on the ImageNet VID dataset. Particularly, our TGBFormer achieves 86.5% mAP while running at around 41.0 FPS on a single Tesla A100 GPU.

Figures (4)

• Figure 1: Pipeline comparison of different methods.
• Figure 2: Framework of the proposed TGBFormer, which builds upon the DETR 26 model.
• Figure 3: Illustration of the proposed STTM.
• Figure 4: Illustration of the proposed STGM.