Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset

Alistair Plum, Laura Bernardy, Tharindu Ranasinghe

TL;DR

This work presents judgeWEL, a Luxembourgish NER dataset created with distant supervision from Wikipedia and Wikidata, followed by LLM-based verification and a human-in-the-loop check. The pipeline yields 28,866 sentences, ~5x larger than prior Luxembourgish NER resources, with improved coverage across entity types and a robust train/dev/test split. Across evaluation, high-end proprietary models (GPT-5 family) achieve the best agreement with human annotators for judging label quality, while transformer encoders (LuxemBERT, mBERT, XLM-R) deliver the strongest NER performance on judgeWEL (F1 > 0.90). Cross-dataset tests show that models trained on judgeWEL transfer well to RTL-NER, indicating reliable label alignment and broader domain coverage, though fully automated labeling with autoregressive LLMs remains challenging for precise token-level labelling. Overall, the study demonstrates a pragmatic, scalable path for building multilingual NER resources in low-resource languages by combining structured knowledge sources with selective LLM verification and limited human oversight.

Abstract

We present judgeWEL, a dataset for named entity recognition (NER) in Luxembourgish, automatically labelled and subsequently verified using large language models (LLM) in a novel pipeline. Building datasets for under-represented languages remains one of the major bottlenecks in natural language processing, where the scarcity of resources and linguistic particularities make large-scale annotation costly and potentially inconsistent. To address these challenges, we propose and evaluate a novel approach that leverages Wikipedia and Wikidata as structured sources of weak supervision. By exploiting internal links within Wikipedia articles, we infer entity types based on their corresponding Wikidata entries, thereby generating initial annotations with minimal human intervention. Because such links are not uniformly reliable, we mitigate noise by employing and comparing several LLMs to identify and retain only high-quality labelled sentences. The resulting corpus is approximately five times larger than the currently available Luxembourgish NER dataset and offers broader and more balanced coverage across entity categories, providing a substantial new resource for multilingual and low-resource NER research.

Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset

TL;DR

This work presents judgeWEL, a Luxembourgish NER dataset created with distant supervision from Wikipedia and Wikidata, followed by LLM-based verification and a human-in-the-loop check. The pipeline yields 28,866 sentences, ~5x larger than prior Luxembourgish NER resources, with improved coverage across entity types and a robust train/dev/test split. Across evaluation, high-end proprietary models (GPT-5 family) achieve the best agreement with human annotators for judging label quality, while transformer encoders (LuxemBERT, mBERT, XLM-R) deliver the strongest NER performance on judgeWEL (F1 > 0.90). Cross-dataset tests show that models trained on judgeWEL transfer well to RTL-NER, indicating reliable label alignment and broader domain coverage, though fully automated labeling with autoregressive LLMs remains challenging for precise token-level labelling. Overall, the study demonstrates a pragmatic, scalable path for building multilingual NER resources in low-resource languages by combining structured knowledge sources with selective LLM verification and limited human oversight.

Abstract

We present judgeWEL, a dataset for named entity recognition (NER) in Luxembourgish, automatically labelled and subsequently verified using large language models (LLM) in a novel pipeline. Building datasets for under-represented languages remains one of the major bottlenecks in natural language processing, where the scarcity of resources and linguistic particularities make large-scale annotation costly and potentially inconsistent. To address these challenges, we propose and evaluate a novel approach that leverages Wikipedia and Wikidata as structured sources of weak supervision. By exploiting internal links within Wikipedia articles, we infer entity types based on their corresponding Wikidata entries, thereby generating initial annotations with minimal human intervention. Because such links are not uniformly reliable, we mitigate noise by employing and comparing several LLMs to identify and retain only high-quality labelled sentences. The resulting corpus is approximately five times larger than the currently available Luxembourgish NER dataset and offers broader and more balanced coverage across entity categories, providing a substantial new resource for multilingual and low-resource NER research.
Paper Structure (26 sections, 4 figures, 4 tables)

This paper contains 26 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Named Entity Recognition
  4. Named Entity Recognition for Low-Resource languages
  5. Language Resources for Luxembourgish
  6. Dataset Construction
  7. Base Data
  8. Entity Linking
  9. First Selection Process
  10. Annotation Improvements
  11. LLM-as-a-Judge
  12. Proprietary Baselines.
  13. Open-Weight Systems.
  14. Instruction-Tuned Systems.
  15. Human-as-a-Judge
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Overview of the JudgeWEL dataset construction pipeline. Gray stages denote automatic processing, green indicates the LLM-based verification stage, and purple marks the final dataset.
  • Figure 2: JSON representation of a labelled sentence used for the dataset.
  • Figure 3: LLM-as-a-Judge prompt used.
  • Figure 4: Model F1 scores trained on judgeWEL or RTL-NER, and evaluated on RTL-NER.