Advancing Multinational License Plate Recognition Through Synthetic and Real Data Fusion: A Comprehensive Evaluation

TL;DR

This work tackles multinational LPR by integrating large-scale synthetic and real data to boost end-to-end license plate recognition. It conducts the most extensive benchmarking to date, evaluating 16 OCR models across 12 diverse datasets and employing three synthetic-data schemes (templates, character permutation, pix2pix) with strong ablations, showing a synergistic benefit when combined. The key finding is that massive synthetic data, even with limited real data, yields state-of-the-art-like performance across intra- and cross-dataset settings, with TRBA plus YOLOv4-CSP and CDCC-NET identifying as a robust end-to-end configuration. The study also provides practical insights into detection, rectification, and speed-accuracy trade-offs, and commits to releasing synthetic data and code to foster reproducibility and broader adoption in multinational LPR contexts.

Abstract

Automatic License Plate Recognition is a frequent research topic due to its wide-ranging practical applications. While recent studies use synthetic images to improve License Plate Recognition (LPR) results, there remain several limitations in these efforts. This work addresses these constraints by comprehensively exploring the integration of real and synthetic data to enhance LPR performance. We subject 16 Optical Character Recognition (OCR) models to a benchmarking process involving 12 public datasets acquired from various regions. Several key findings emerge from our investigation. Primarily, the massive incorporation of synthetic data substantially boosts model performance in both intra- and cross-dataset scenarios. We examine three distinct methodologies for generating synthetic data: template-based generation, character permutation, and utilizing a Generative Adversarial Network (GAN) model, each contributing significantly to performance enhancement. The combined use of these methodologies demonstrates a notable synergistic effect, leading to end-to-end results that surpass those reached by state-of-the-art methods and established commercial systems. Our experiments also underscore the efficacy of synthetic data in mitigating challenges posed by limited training data, enabling remarkable results to be achieved even with small fractions of the original training data. Finally, we investigate the trade-off between accuracy and speed among different models, identifying those that strike the optimal balance in each intra-dataset and cross-dataset settings.

Figures (9)

• Figure 1: Examples of the template-based lp images we created for this work. Notably, any sequence can be generated for each template. The background and character images were gathered from the internet$^1$. During training, these LP images are subjected to various transformations to introduce variability.
• Figure 2: Some lp images created by permuting the positions of the characters within each lp and then applying transformations. The images in the top row are the originals, while the others were synthesized.
• Figure 3: Examples of image pairs used for training the pix2pix model. To create the input masks, labels are required for both the lp's layout and corners, as well as for the bounding box of each character.
• Figure 4: Examples of selected images from those generated using pix2pix. From top to bottom, we show American, Brazilian, Chinese, European, Mercosur, and Taiwanese lp.
• Figure 5: Representative lp from the datasets used in our experiments, highlighting the diversity of lp layouts and visual characteristics. Adapted from laroca2023leveraging.