Veri-Car: Towards Open-world Vehicle Information Retrieval

TL;DR

Veri-Car tackles open-world vehicle information retrieval by decomposing the problem into two retrieval streams (Make/Type/Model/Year and Color) and a robust license plate pipeline. It combines pre-trained vision encoders with metric-learning losses, notably a hierarchical Multi-Similarity Loss, to create embeddings that generalize to unseen car models. The system integrates OOD detection to trigger human-in-the-loop labeling for new classes and leverages synthetic license plate data to achieve multinational LPR Generalization, outperformingcountry-specific baselines. Across color, model-year, and license plate tasks, Veri-Car demonstrates strong open-world performance, effective OOD detection, and practical end-to-end accuracy, highlighting its potential for real-world deployment in tolling, security, and automotive services.

Abstract

Many industrial and service sectors require tools to extract vehicle characteristics from images. This is a complex task not only by the variety of noise, and large number of classes, but also by the constant introduction of new vehicle models to the market. In this paper, we present Veri-Car, an information retrieval integrated approach designed to help on this task. It leverages supervised learning techniques to accurately identify the make, type, model, year, color, and license plate of cars. The approach also addresses the challenge of handling open-world problems, where new car models and variations frequently emerge, by employing a sophisticated combination of pre-trained models, and a hierarchical multi-similarity loss. Veri-Car demonstrates robust performance, achieving high precision and accuracy in classifying both seen and unseen data. Additionally, it integrates an ensemble license plate detection, and an OCR model to extract license plate numbers with impressive accuracy.

Figures (12)

• Figure 1: Veri-Car Architecture.
• Figure 2: Cars labels hierarchies.
• Figure 3: Retrieval flow chart.
• Figure 4: (a) Change in Precision@1 when new classes are added into the retrieval database. (b) Precision@1 relative to number of samples per class in the database.
• Figure 5: Multi-similarity loss color confusion matrix. Each cell reports the percentage of observations falling in the combination of predicted vs true labels.