Evaluation of LLMs on Long-tail Entity Linking in Historical Documents

TL;DR

This paper investigates long-tail entity linking in historical documents by comparing large language models (GPT-3.5 and Llama-3) against a state-of-the-art EL framework (ReLiK) on the MHERCL v0.1.2 benchmark. It treats EL as a sequence-to-sequence task via one-shot JSON prompts that map text spans to Wikipedia page titles, using Wikipedia as the identifier to avoid QID hallucinations. Results show that large LLMs achieve higher recall than ReLiK, particularly for rare entities, though precision suffers due to over-generation; overall, LLMs demonstrate strong potential to retrieve long-tail entities and may serve as effective retrievers or augmenters in hybrid EL systems. The study highlights OCR noise and contextual limitations in historical texts, and suggests future work in prompting strategies, in-context learning, knowledge injection, and hybrid architectures to balance recall and precision in long-tail EL scenarios.

Abstract

Entity Linking (EL) plays a crucial role in Natural Language Processing (NLP) applications, enabling the disambiguation of entity mentions by linking them to their corresponding entries in a reference knowledge base (KB). Thanks to their deep contextual understanding capabilities, LLMs offer a new perspective to tackle EL, promising better results than traditional methods. Despite the impressive generalization capabilities of LLMs, linking less popular, long-tail entities remains challenging as these entities are often underrepresented in training data and knowledge bases. Furthermore, the long-tail EL task is an understudied problem, and limited studies address it with LLMs. In the present work, we assess the performance of two popular LLMs, GPT and LLama3, in a long-tail entity linking scenario. Using MHERCL v0.1, a manually annotated benchmark of sentences from domain-specific historical texts, we quantitatively compare the performance of LLMs in identifying and linking entities to their corresponding Wikidata entries against that of ReLiK, a state-of-the-art Entity Linking and Relation Extraction framework. Our preliminary experiments reveal that LLMs perform encouragingly well in long-tail EL, indicating that this technology can be a valuable adjunct in filling the gap between head and long-tail EL.

Figures (1)

• Figure 1: Measurement of the Entity Linking F1 and Recall score Across Different Entity Occurrence Thresholds for all the employed models