Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot Vehicle Model Recognition via Text-Based Retrieval-Augmented Generation

Wei-Chia Chang, Yan-Ann Chen

TL;DR

This work tackles zero-shot vehicle make–model recognition (VMMR) by moving recognition from the image domain to text. It introduces a retrieval-augmented generation (RAG) pipeline that first converts each vehicle image into a descriptive text via a vision–language model, then retrieves related textual entries from a dynamic database and uses a language model to reason over the combination to predict the make and model. The authors demonstrate that this text-centric approach, which grounds LM reasoning in retrieved evidence, substantially outperforms a CLIP baseline on 10 newly released car models, achieving about $0.37$ accuracy versus $0.21$ for CLIP (roughly a 16 percentage point gain). The method is lightweight, relies on open-source models, and supports rapid updates by simply adding textual descriptions of new vehicles, making it practical for scalable, edge-ready VMMR in smart-city contexts.

Abstract

Vehicle make and model recognition (VMMR) is an important task in intelligent transportation systems, but existing approaches struggle to adapt to newly released models. Contrastive Language-Image Pretraining (CLIP) provides strong visual-text alignment, yet its fixed pretrained weights limit performance without costly image-specific finetuning. We propose a pipeline that integrates vision language models (VLMs) with Retrieval-Augmented Generation (RAG) to support zero-shot recognition through text-based reasoning. A VLM converts vehicle images into descriptive attributes, which are compared against a database of textual features. Relevant entries are retrieved and combined with the description to form a prompt, and a language model (LM) infers the make and model. This design avoids large-scale retraining and enables rapid updates by adding textual descriptions of new vehicles. Experiments show that the proposed method improves recognition by nearly 20% over the CLIP baseline, demonstrating the potential of RAG-enhanced LM reasoning for scalable VMMR in smart-city applications.

Zero-Shot Vehicle Model Recognition via Text-Based Retrieval-Augmented Generation

TL;DR

This work tackles zero-shot vehicle make–model recognition (VMMR) by moving recognition from the image domain to text. It introduces a retrieval-augmented generation (RAG) pipeline that first converts each vehicle image into a descriptive text via a vision–language model, then retrieves related textual entries from a dynamic database and uses a language model to reason over the combination to predict the make and model. The authors demonstrate that this text-centric approach, which grounds LM reasoning in retrieved evidence, substantially outperforms a CLIP baseline on 10 newly released car models, achieving about accuracy versus for CLIP (roughly a 16 percentage point gain). The method is lightweight, relies on open-source models, and supports rapid updates by simply adding textual descriptions of new vehicles, making it practical for scalable, edge-ready VMMR in smart-city contexts.

Abstract

Vehicle make and model recognition (VMMR) is an important task in intelligent transportation systems, but existing approaches struggle to adapt to newly released models. Contrastive Language-Image Pretraining (CLIP) provides strong visual-text alignment, yet its fixed pretrained weights limit performance without costly image-specific finetuning. We propose a pipeline that integrates vision language models (VLMs) with Retrieval-Augmented Generation (RAG) to support zero-shot recognition through text-based reasoning. A VLM converts vehicle images into descriptive attributes, which are compared against a database of textual features. Relevant entries are retrieved and combined with the description to form a prompt, and a language model (LM) infers the make and model. This design avoids large-scale retraining and enables rapid updates by adding textual descriptions of new vehicles. Experiments show that the proposed method improves recognition by nearly 20% over the CLIP baseline, demonstrating the potential of RAG-enhanced LM reasoning for scalable VMMR in smart-city applications.
Paper Structure (17 sections, 4 equations, 4 figures, 5 tables)

This paper contains 17 sections, 4 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Method Design
  3. Problem Formulation
  4. Pipeline Design
  5. Experimental Results
  6. Dataset Introduction
  7. Experimental Setup
  8. Experimental Procedure
  9. Establishing a Baseline
  10. Building the Retrieval Database
  11. Recognizing Vehicle Models with RAG Integration
  12. Calculating Model Performance
  13. Performance Evaluation
  14. Varying K Values
  15. Investigating Prediction Errors in Image Comparison
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Experimental Framework for Car Model Recognition: CLIP vs. RAG Approach
  • Figure 2: Confusion Matrices for Car Model Classification: CLIP vs. RAG Approach
  • Figure 3: Retrieval Rank Distribution of True Models in Top-5 RAG Results
  • Figure 4: EZ6 and Spectre: A Comparative View