ParsiPy: NLP Toolkit for Historical Persian Texts in Python

TL;DR

The paper tackles the lack of NLP resources for Pārsīg, a historical Persian language, by introducing ParsiPy, a Python toolkit offering tokenization, lemmatization, POS tagging, phoneme-to-transliteration, and embeddings tailored to Parsig. It demonstrates the toolkit on the Parsig Database, evaluating P2T, lemmatization, and POS tagging, and shows competitive performance with a modular, rule-based plus statistical approach suitable for low-resource settings. The work provides a pipeline API for consistent processing and transliteration between phonemic representations and Parsig script, contributing to computational philology and digital preservation of ancient texts. It also lays groundwork for extending these methods to other historical languages through open-source tooling and future Unicode standardization for Parsig scripts.

Abstract

The study of historical languages presents unique challenges due to their complex orthographic systems, fragmentary textual evidence, and the absence of standardized digital representations of text in those languages. Tackling these challenges needs special NLP digital tools to handle phonetic transcriptions and analyze ancient texts. This work introduces ParsiPy, an NLP toolkit designed to facilitate the analysis of historical Persian languages by offering modules for tokenization, lemmatization, part-of-speech tagging, phoneme-to-transliteration conversion, and word embedding. We demonstrate the utility of our toolkit through the processing of Parsig (Middle Persian) texts, highlighting its potential for expanding computational methods in the study of historical languages. Through this work, we contribute to computational philology, offering tools that can be adapted for the broader study of ancient texts and their digital preservation.

Figures (6)

• Figure 1: An example of Pārsīg text in original written form from Andarze Azarabade Mehrsepandan database. It reads "ān uzīd frāmōš kun ud ān nē mad ēstēd rāy tēmār bēš ma bar" and it means "Forget what is gone and do not worry about what has not yet come."
• Figure 2: Parsipy Framework Overview. Input string $s$ goes into tokenized into $n$ tokens ($t_1, \cdots,t_n$) and the embedding module would generate word embeddings for each token ($v_1, \cdots, v_n$). Lemmatizer extracts the lemma for each token ($l_1,\cdots,l_n$), POSTagger tags each token with its part-of-speech in the sentence ($p_1, \cdots,p_n$), and Transliteration module (Phoneme to Transliteration: P2T) generates a middle-form representation of tokens by which they can transform into Parsig in hand-written form. The example sentence is from Corpus Of Pahlavi Texts (Jamaspji Dastur Minochehrji Jamasp Asana) which is gathered and translated by Goshtasb2022Justice. The English translation of it is "It ended with greetings (= happiness)" and it is chosen for the sake of simplicity.
• Figure 3: The 14 letters of the Pārsīg alphabet, used in the Middle Persian language.
• Figure 4: Confusion matrix for HMM & Viterbi
• Figure 5: Confusion matrix for Logistic regression