Shifting NER into High Gear: The Auto-AdvER Approach

TL;DR

Auto-AdvER introduces a domain-specific NER schema for English car advertisements with three labels (Condition, Historic, Sales Options) and presents a 605-ad, 104,382-token dataset developed through industry–academic collaboration using a DevOps-inspired annotation process that achieved an inter-annotator $F1$-score of $0.92$. The study compares encoder-based models (BERT, DeBERTaV3) against large decoder LLMs (GPT-4o, Gemini, Llama, Qwen) under in-context learning, finding that LLMs generally outperform encoders but incur higher cost and imperfect accuracy. Key contributions include a novel annotation framework, a publicly shareable domain dataset, and a critical discussion of practical deployment in automotive analytics and customer insights, with future work on entity linking and broader-domain applicability. The work lays groundwork for richer ad analytics, informing market dynamics, maintenance forecasting, and consumer safety efforts in the automotive sector. Overall, Auto-AdvER provides a benchmark for domain-specific NER in automotive text and a blueprint for extending annotation schemas to other specialist domains.

Abstract

This paper presents a case study on the development of Auto-AdvER, a specialised named entity recognition schema and dataset for text in the car advertisement genre. Developed with industry needs in mind, Auto-AdvER is designed to enhance text mining analytics in this domain and contributes a linguistically unique NER dataset. We present a schema consisting of three labels: "Condition", "Historic" and "Sales Options". We outline the guiding principles for annotation, describe the methodology for schema development, and show the results of an annotation study demonstrating inter-annotator agreement of 92% F1-Score. Furthermore, we compare the performance by using encoder-only models: BERT, DeBERTaV3 and decoder-only open and closed source Large Language Models (LLMs): Llama, Qwen, GPT-4 and Gemini. Our results show that the class of LLMs outperforms the smaller encoder-only models. However, the LLMs are costly and far from perfect for this task. We present this work as a stepping stone toward more fine-grained analysis and discuss Auto-AdvER's potential impact on advertisement analytics and customer insights, including applications such as the analysis of market dynamics and data-driven predictive maintenance. Our schema, as well as our associated findings, are suitable for both private and public entities considering named entity recognition in the automotive domain, or other specialist domains.

Figures (1)

• Figure 1: An example of the annotation interface. It is a customised NER annotation using Prodigy.