VFM-Det: Towards High-Performance Vehicle Detection via Large Foundation Models

TL;DR

VFM-Det introduces a vehicle-focused detection paradigm that exploits a pre-trained VehicleMAE backbone and a large language model to enhance region proposals and semantic understanding. A novel VAtt2Vec module bridges the semantic attributes and vision features via contrastive learning, yielding unified attribute representations that align with visual cues. Across Cityscapes, UA-DETRAC, and COCO2017 vehicle subsets, VFM-Det consistently outperforms strong baselines, with notable gains over Mask R-CNN and competitive SOTA detectors. The work demonstrates the practical value of task-specific foundation models and cross-modal alignment for high-performance vehicle detection, while also acknowledging computational overhead and non-end-to-end training as areas for future improvement.

Abstract

Existing vehicle detectors are usually obtained by training a typical detector (e.g., YOLO, RCNN, DETR series) on vehicle images based on a pre-trained backbone (e.g., ResNet, ViT). Some researchers also exploit and enhance the detection performance using pre-trained large foundation models. However, we think these detectors may only get sub-optimal results because the large models they use are not specifically designed for vehicles. In addition, their results heavily rely on visual features, and seldom of they consider the alignment between the vehicle's semantic information and visual representations. In this work, we propose a new vehicle detection paradigm based on a pre-trained foundation vehicle model (VehicleMAE) and a large language model (T5), termed VFM-Det. It follows the region proposal-based detection framework and the features of each proposal can be enhanced using VehicleMAE. More importantly, we propose a new VAtt2Vec module that predicts the vehicle semantic attributes of these proposals and transforms them into feature vectors to enhance the vision features via contrastive learning. Extensive experiments on three vehicle detection benchmark datasets thoroughly proved the effectiveness of our vehicle detector. Specifically, our model improves the baseline approach by $+5.1\%$, $+6.2\%$ on the $AP_{0.5}$, $AP_{0.75}$ metrics, respectively, on the Cityscapes dataset.The source code of this work will be released at https://github.com/Event-AHU/VFM-Det.

Figures (7)

• Figure 1: Illustration of our proposed the Large Vision Model (VehicleMAE wang2023structural) and Large Language Model (T5 raffel2020exploring) based vehicle detection.
• Figure 2: An overview of our proposed VFM-Det framework which exploits vision feature enhancement and semantic-vision alignment for high-performance vehicle detection using pre-trained foundation models. On the basis of the region proposal-based detectors, we introduce the VehicleMAE encoder to enhance the vision features. Meanwhile, we predict the vehicle attributes and learn a unified attribute representation via VAtt2Vec, to achieve alignment between the vision and semantic high-level features. More details about the VAtt2Vec module can be found in Fig. \ref{['fig:VAtt2Vec']}. Extensive experiments on three vehicle detection benchmark datasets thoroughly proved the effectiveness of our proposed strategies.
• Figure 3: The detailed architecture of our proposed VAtt2Vec module.
• Figure 4: Structural diagram of GRU module.
• Figure 5: Visualization of the detection results from Mask R-CNN and VFM-Det.