Vehicle-centric Perception via Multimodal Structured Pre-training

TL;DR

This work introduces VehicleMAE-V2, a vehicle-centric pre-trained large model that embeds symmetry-guided masking, contour-guided structural reconstruction, and semantics-guided cross-modal learning to produce generalized vehicle representations. Leveraging the Autobot4M dataset (≈4.02M images and 12,693 texts), the method achieves state-of-the-art or strong performance across five downstream tasks—VAR, V-Det, V-Reid, VFR, and VPS—demonstrating the value of structured priors in pre-training. Ablation and efficiency analyses substantiate the contributions of each module and the data-scale benefits, indicating practical impact for real-world vehicle perception systems. The work also outlines limitations and avenues for future work, including broader modality integration and more detection-friendly pre-training strategies.

Abstract

Vehicle-centric perception plays a crucial role in many intelligent systems, including large-scale surveillance systems, intelligent transportation, and autonomous driving. Existing approaches lack effective learning of vehicle-related knowledge during pre-training, resulting in poor capability for modeling general vehicle perception representations. To handle this problem, we propose VehicleMAE-V2, a novel vehicle-centric pre-trained large model. By exploring and exploiting vehicle-related multimodal structured priors to guide the masked token reconstruction process, our approach can significantly enhance the model's capability to learn generalizable representations for vehicle-centric perception. Specifically, we design the Symmetry-guided Mask Module (SMM), Contour-guided Representation Module (CRM) and Semantics-guided Representation Module (SRM) to incorporate three kinds of structured priors into token reconstruction including symmetry, contour and semantics of vehicles respectively. SMM utilizes the vehicle symmetry constraints to avoid retaining symmetric patches and can thus select high-quality masked image patches and reduce information redundancy. CRM minimizes the probability distribution divergence between contour features and reconstructed features and can thus preserve holistic vehicle structure information during pixel-level reconstruction. SRM aligns image-text features through contrastive learning and cross-modal distillation to address the feature confusion caused by insufficient semantic understanding during masked reconstruction. To support the pre-training of VehicleMAE-V2, we construct Autobot4M, a large-scale dataset comprising approximately 4 million vehicle images and 12,693 text descriptions. Extensive experiments on five downstream tasks demonstrate the superior performance of VehicleMAE-V2.

Figures (10)

• Figure 1: Comparison between existing large models and our newly proposed VehicleMAE-V2 on five downstream vehicle related tasks.
• Figure 2: An illustration of our newly proposed pre-trained vehicle-centric large model VehicleMAE-V2, which is pre-trained on 4M vehicle images based on structural and semantic priors. Five downstream tasks are evaluated to validate the effectiveness and generalization of the proposed VehicleMAE-V2.
• Figure 3: An overview of our proposed structured pre-training framework for vehicle perception, termed VehicleMAE-V2. The framework is built upon the structural and semantic information of vehicle targets and consists of three key modules: a Symmetry-guided Mask Module that exploits vehicle symmetry priors to guide the masking strategy; a Contour-guided Representation Module that explicitly models and constrains the spatial structural representation of vehicles; and a Semantics-guided Representation Module that aligns visual features with semantic information to enhance the model’s semantic understanding capability.
• Figure 4: Comparison of different masked strategies used in our VehicleMAE-V2 framework, i.e., (b). the random masking, (c). the box-guided masking, and (d). the symmetry-guided masking. The blue dotted line represents the box of the vehicle target. The red straight line is the axis of symmetry of the vehicle. The green box and double-headed arrow dashed line indicate the unmasked symmetric patch in the box-guide mask strategy. The yellow box and single-arrow dashed line indicate the result after eliminating symmetrical blocks in the symmetry-guide mask strategy.
• Figure 5: Representative samples of our proposed Autobot4M dataset.