Modern Models, Medieval Texts: A POS Tagging Study of Old Occitan

TL;DR

This work provides the first systematic evaluation of open-source large language models for part-of-speech tagging in Old Occitan, a highly non-standardized and low-resource historical Romance language. By comparing eight models across two diverse corpora (hagiographical Vida de Sant Honorat and the medical Albucasis text) and employing zero-shot, Prompt A, and Prompt B prompting strategies, the study reveals significant limitations due to orthographic variation and sparse training data, with larger models showing partial robustness and cross-lingual transfer offering notable advantages in some configurations. The authors deliver a detailed error analysis and practical recommendations for preprocessing, prompting, and model selection, and release a new Old Occitan POS tagging dataset to foster reproducibility and future research in historical NLP. Overall, the work informs both computational linguistics and digital humanities about the current capabilities and necessary developments for processing challenging historical languages.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing, yet their effectiveness in handling historical languages remains largely unexplored. This study examines the performance of open-source LLMs in part-of-speech (POS) tagging for Old Occitan, a historical language characterized by non-standardized orthography and significant diachronic variation. Through comparative analysis of two distinct corpora-hagiographical and medical texts-we evaluate how current models handle the inherent challenges of processing a low-resource historical language. Our findings demonstrate critical limitations in LLM performance when confronted with extreme orthographic and syntactic variability. We provide detailed error analysis and specific recommendations for improving model performance in historical language processing. This research advances our understanding of LLM capabilities in challenging linguistic contexts while offering practical insights for both computational linguistics and historical language studies.

Figures (9)

• Figure 1: (a) Geographic distribution of Old Occitan with its principal dialect zones sibille:2024. (b) Orthographic diversity in Old Occitan texts, as evidenced by multiple graphical variants of the same term, illustrating inherent challenges for modern LLMs.
• Figure 2: Part-of-Speech distribution for both texts: Albucasis (blue) and Vida de Sant Honorat (red).
• Figure 3: Accuracy distribution across different prompting strategies and datasets.
• Figure 4: Accuracy heatmap for models and prompting strategies. Results on the left correspond to the NAF6195 dataset and on the right to Albucasis.
• Figure 5: Accuracy across phrases vs. Prompt Strategies for the NAF6195 and Albucasis datasets.