LP-LLM: End-to-End Real-World Degraded License Plate Text Recognition via Large Multimodal Models

TL;DR

This work tackles real-world license plate recognition under severe degradations by shifting from a restoration-first pipeline to end-to-end structured multimodal reasoning. It introduces the Character-Aware Multimodal Reasoning Module (CMRM) that assigns $K$ learnable slot queries to plate positions and uses slot-to-vision cross-attention to extract position-specific evidence, followed by residual injection into visual embeddings for autoregressive generation with a large vision-language model (Qwen3-VL). The model is trained with parameter-efficient LoRA fine-tuning, updating only the LoRA adapters and CMRM while keeping the backbone fixed, enabling domain adaptation without sacrificing generalization. Empirical results on synthetic and real degraded data show large gains over restoration-based and general VLM baselines, achieving state-of-the-art accuracy (≈89.4%) and low CER while maintaining practical latency. By encoding explicit character structure priors, the approach demonstrates robust, end-to-end license plate recognition in challenging conditions and highlights the value of structured priors in large multimodal models for low-quality text tasks.

Abstract

Real-world License Plate Recognition (LPR) faces significant challenges from severe degradations such as motion blur, low resolution, and complex illumination. The prevailing "restoration-then-recognition" two-stage paradigm suffers from a fundamental flaw: the pixel-level optimization objectives of image restoration models are misaligned with the semantic goals of character recognition, leading to artifact interference and error accumulation. While Vision-Language Models (VLMs) have demonstrated powerful general capabilities, they lack explicit structural modeling for license plate character sequences (e.g., fixed length, specific order). To address this, we propose an end-to-end structure-aware multimodal reasoning framework based on Qwen3-VL. The core innovation lies in the Character-Aware Multimodal Reasoning Module (CMRM), which introduces a set of learnable Character Slot Queries. Through a cross-attention mechanism, these queries actively retrieve fine-grained evidence corresponding to character positions from visual features. Subsequently, we inject these character-aware representations back into the visual tokens via residual modulation, enabling the language model to perform autoregressive generation based on explicit structural priors. Furthermore, combined with the LoRA parameter-efficient fine-tuning strategy, the model achieves domain adaptation while retaining the generalization capabilities of the large model. Extensive experiments on both synthetic and real-world severely degraded datasets demonstrate that our method significantly outperforms existing restoration-recognition combinations and general VLMs, validating the superiority of incorporating structured reasoning into large models for low-quality text recognition tasks.