DANSK and DaCy 2.6.0: Domain Generalization of Danish Named Entity Recognition

TL;DR

Danish NER has suffered from scarce open, fine-grained datasets and limited cross-domain evaluation. The authors introduce DANSK, a 18-label, OntoNotes-like NER dataset drawn from the Danish Gigaword Corpus, and DaCy 2.6.0 with three generalizable fine-grained models trained on DANSK, accompanied by a cross-domain evaluation of SOTA Danish models. Through automated and manual annotation refinement, DANSK achieves high inter-annotator agreement, while DaCy’s large model attains macro F1 around 0.82 across 18 entity types; results reveal persistent domain-specific performance gaps that highlight generalization challenges. The work provides a publicly available resource and evaluation framework for Danish NER, enabling more robust cross-domain research and guiding future improvements in domain generalization for Danish NLP.

Abstract

Named entity recognition is one of the cornerstones of Danish NLP, essential for language technology applications within both industry and research. However, Danish NER is inhibited by a lack of available datasets. As a consequence, no current models are capable of fine-grained named entity recognition, nor have they been evaluated for potential generalizability issues across datasets and domains. To alleviate these limitations, this paper introduces: 1) DANSK: a named entity dataset providing for high-granularity tagging as well as within-domain evaluation of models across a diverse set of domains; 2) DaCy 2.6.0 that includes three generalizable models with fine-grained annotation; and 3) an evaluation of current state-of-the-art models' ability to generalize across domains. The evaluation of existing and new models revealed notable performance discrepancies across domains, which should be addressed within the field. Shortcomings of the annotation quality of the dataset and its impact on model training and evaluation are also discussed. Despite these limitations, we advocate for the use of the new dataset DANSK alongside further work on the generalizability within Danish NER.

Figures (4)

• Figure 1: The decision tree for automated conflict resolvement of multi-annotated texts. Each annotation span in a text followed the steps from 1 to 4 on the diagram. The decision tree was only followed for annotation spans found in texts that had been annotated by at least four raters.
• Figure 2: An example of a text along with its four annotations being processed on the basis of the decision-tree in Figure \ref{['fig:automated_conflict_resolvement_rules']}.
• Figure 3: Confusion matrix across annotated tokens before and after the automated streamlining.
• Figure 4: Figure displaying the domain performance in macro F1-scores of the three models on the test partition of DANSK. The size of the circles represents the size of the domains, and thus their relative weighted impact on the overall scores. See Table \ref{['tab:ner_domain']} for scores.