CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents

TL;DR

A human-annotated benchmark based on Czech historical documents is introduced, containing human-defined topics together with manually annotated spans and supporting evaluation at both document and word levels, revealing substantial variability among LLMs.

Abstract

Topic localization aims to identify spans of text that express a given topic defined by a name and description. To study this task, we introduce a human-annotated benchmark based on Czech historical documents, containing human-defined topics together with manually annotated spans and supporting evaluation at both document and word levels. Evaluation is performed relative to human agreement rather than a single reference annotation. We evaluate a diverse range of large language models alongside BERT-based models fine-tuned on a distilled development dataset. Results reveal substantial variability among LLMs, with performance ranging from near-human topic detection to pronounced failures in span localization. While the strongest models approach human agreement, the distilled token embedding models remain competitive despite their smaller scale. The dataset and evaluation framework are publicly available at: https://github.com/dcgm/czechtopic.

Figures (6)

• Figure 1: Illustration of the topic localization task. A topic is defined by a name and a textual description. The objective is to identify and localize all spans in the document that correspond to the given topic.
• Figure 2: Example from our topic localization dataset showing a text annotated with spans belonging to two distinct topics.
• Figure 3: Distribution statistics of our topic localization dataset. Distribution of the number of texts in which a topic appears in Phase 1 (a) and Phase 2 (b). The Phase 2 distribution is aggregated across all annotators. Note that some topics are not annotated by certain annotators, resulting in zero occurrences for those annotator--topic pairs. (c) Distribution of annotated span lengths measured in words for Phase 2 annotations.
• Figure 4: Overview of the cross-encoder architecture used for topic localization. The topic and text are jointly encoded by BERT. A similarity matrix is computed between topic tokens and text tokens, and each text token receives a score given by the maximum similarity across all topic tokens.
• Figure 5: Comparison of model performance. The top row shows word-level F1 scores and the bottom row text-level F1 scores, each with 95% confidence intervals. (a) Large language models and (b) BERT-based models. The human baseline is highlighted in red. In (b), the teacher model is highlighted in green.