Temporal Object-Aware Vision Transformer for Few-Shot Video Object Detection

TL;DR

This work tackles Few-Shot Video Object Detection (FSVOD), where the goal is to detect novel object classes in video with limited labeled examples while maintaining temporal consistency. It introduces an object-aware temporal fusion framework built on a language-aligned vision encoder (OWL-ViT), a temporal frame decoder that selectively propagates high-confidence object features across frames, and few-shot detection heads that use support prototypes and cosine similarity for robust recognition. Key contributions include the object-aware temporal propagation mechanism, the integration of vision-language pretraining for open-set generalization, and extensive evaluations showing state-of-the-art AP gains across four benchmarks (FSVOD-500, FSYTV-40, VidOR, VidVRD) in the 5-shot setting, along with analyses of thresholding and efficiency. The results demonstrate improved temporal reasoning, reduced noise accumulation, and strong generalization to unseen categories, with practical impact for real-time video analysis in low-label regimes; code is publicly available.

Abstract

Few-shot Video Object Detection (FSVOD) addresses the challenge of detecting novel objects in videos with limited labeled examples, overcoming the constraints of traditional detection methods that require extensive training data. This task presents key challenges, including maintaining temporal consistency across frames affected by occlusion and appearance variations, and achieving novel object generalization without relying on complex region proposals, which are often computationally expensive and require task-specific training. Our novel object-aware temporal modeling approach addresses these challenges by incorporating a filtering mechanism that selectively propagates high-confidence object features across frames. This enables efficient feature progression, reduces noise accumulation, and enhances detection accuracy in a few-shot setting. By utilizing few-shot trained detection and classification heads with focused feature propagation, we achieve robust temporal consistency without depending on explicit object tube proposals. Our approach achieves performance gains, with AP improvements of 3.7% (FSVOD-500), 5.3% (FSYTV-40), 4.3% (VidOR), and 4.5 (VidVRD) in the 5-shot setting. Further results demonstrate improvements in 1-shot, 3-shot, and 10-shot configurations. We make the code public at: https://github.com/yogesh-iitj/fs-video-vit

Figures (6)

• Figure 1: Given a support set containing novel objects and a target video, the goal of FSVODfsvod is to detect all the instances of novel objects on the target video. We propose an object-aware temporally consistent few-shot object detection framework that significantly improves the state of the art for this task.
• Figure 2: Object-aware Temporal Consistency. We utilize the OWL-ViT encoder output tokens as queries but, critically, only forward the matched object tokens ($c^p > \tau$) as key-value pairs to the next frame's decoder. This mechanism selectively propagates high-confidence object features across frames. This enables efficient feature progression, reducing noise accumulation. We empirically found a threshold ($\tau$) and compared it with class probability ($c^p$) scores for selective propagation. This selective propagation mechanism significantly reduces noise accumulation across frames and maintains focused, consistent visual representations of detected objects throughout the video sequence.
• Figure 3: Few-shot Classification and Detection Heads. Our architecture processes object queries through parallel projection heads for classification and localization. Classification embeddings are compared against support embeddings via cosine similarity, while the detection head predicts bounding box coordinates. The temporal consistency is maintained by propagating matched object queries across frames.
• Figure 4: Threshold effect ($c^p > \tau$) during the object-aware temporal consistency.
• Figure 5: Qualitative comparison FSVOD-500 test set videos. Our method accurately detects multiple visually distinct instances of Bactrian camels (top) and precisely localizes the Autogyro (bottom), whereas FSVOD misses several camel instances.