Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Solution for the CVPR 2023 1st foundation model challenge-Track2

Haonan Xu, Yurui Huang, Sishun Pan, Zhihao Guan, Yi Xu, Yang Yang

TL;DR

This paper tackles cross-modal, cross-domain retrieval for traffic imagery by splitting the task into pedestrian and vehicle sub-tasks and leveraging specialized base models (IRRA for pedestrians, BLIP for vehicles). The pedestrian branch introduces multi-label Attribute Classification and Inclusion Relation Matching to improve fine-grained alignment between image and text. The vehicle branch employs color patch augmentation and fine-grained image-text contrastive learning to enhance color-aware matching. Ablation studies and final results on the CVPR 2023 Track2 B-board show strong gains, with a final score of 70.9, demonstrating the effectiveness of attribute supervision and targeted augmentations for cross-modal retrieval in transportation scenarios.

Abstract

In this paper, we propose a solution for cross-modal transportation retrieval. Due to the cross-domain problem of traffic images, we divide the problem into two sub-tasks of pedestrian retrieval and vehicle retrieval through a simple strategy. In pedestrian retrieval tasks, we use IRRA as the base model and specifically design an Attribute Classification to mine the knowledge implied by attribute labels. More importantly, We use the strategy of Inclusion Relation Matching to make the image-text pairs with inclusion relation have similar representation in the feature space. For the vehicle retrieval task, we use BLIP as the base model. Since aligning the color attributes of vehicles is challenging, we introduce attribute-based object detection techniques to add color patch blocks to vehicle images for color data augmentation. This serves as strong prior information, helping the model perform the image-text alignment. At the same time, we incorporate labeled attributes into the image-text alignment loss to learn fine-grained alignment and prevent similar images and texts from being incorrectly separated. Our approach ranked first in the final B-board test with a score of 70.9.

The Solution for the CVPR 2023 1st foundation model challenge-Track2

TL;DR

This paper tackles cross-modal, cross-domain retrieval for traffic imagery by splitting the task into pedestrian and vehicle sub-tasks and leveraging specialized base models (IRRA for pedestrians, BLIP for vehicles). The pedestrian branch introduces multi-label Attribute Classification and Inclusion Relation Matching to improve fine-grained alignment between image and text. The vehicle branch employs color patch augmentation and fine-grained image-text contrastive learning to enhance color-aware matching. Ablation studies and final results on the CVPR 2023 Track2 B-board show strong gains, with a final score of 70.9, demonstrating the effectiveness of attribute supervision and targeted augmentations for cross-modal retrieval in transportation scenarios.

Abstract

In this paper, we propose a solution for cross-modal transportation retrieval. Due to the cross-domain problem of traffic images, we divide the problem into two sub-tasks of pedestrian retrieval and vehicle retrieval through a simple strategy. In pedestrian retrieval tasks, we use IRRA as the base model and specifically design an Attribute Classification to mine the knowledge implied by attribute labels. More importantly, We use the strategy of Inclusion Relation Matching to make the image-text pairs with inclusion relation have similar representation in the feature space. For the vehicle retrieval task, we use BLIP as the base model. Since aligning the color attributes of vehicles is challenging, we introduce attribute-based object detection techniques to add color patch blocks to vehicle images for color data augmentation. This serves as strong prior information, helping the model perform the image-text alignment. At the same time, we incorporate labeled attributes into the image-text alignment loss to learn fine-grained alignment and prevent similar images and texts from being incorrectly separated. Our approach ranked first in the final B-board test with a score of 70.9.
Paper Structure (8 sections, 9 equations, 3 figures, 1 table)

This paper contains 8 sections, 9 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overall Architecture
  5. Pedestrian Retrieval
  6. Vehicle Retrieval
  7. Experiments
  8. Conclusion

Figures (3)

  • Figure 1: This example illustrates the improvement of using a visual prompt augmentation strategy.
  • Figure 2: The ground truth text of the image is described as "A man wearing glasses and long sleeves.", for which both our model and IRRA are correctly retrieved. However, when the text is "A man wearing glasses.", our model can retrieve the image, while IRRA cannot.
  • Figure 3: Overall Architecture. On the left of the figure is the pedestrian model architecture. On the basis of IRRA, Attribute Classification, and Inclusion Relation Matching are innovatively added. The design of the model fully mines the data annotation information and makes the performance of image-text pairs more excellent in fine-grained matching. The right side of the figure shows the vehicle model architecture. We adopt BLIP as a framework to optimize the fine-grained matching of image-text pairs and mine difficult samples. We also introduce a visual prompt augmentation strategy to give the model prior knowledge to help the model better recognize the vehicle color.