A Simple Joint Model for Improved Contextual Neural Lemmatization

TL;DR

The paper tackles context-sensitive lemmatization by jointly modeling morphological tagging and lemmatization in a simple neural framework. It combines a character-aware tagger with a hard-attention lemmatizer and uses jackknifing to mitigate exposure bias, with decoding options including greedy and crunching. Empirically, the approach achieves state-of-the-art lemmatization accuracy across 20 UD languages, and analyses reveal that benefits are largest for morphologically rich or low-resource languages; gold-tag signals hint at substantial upper-bound gains. The work demonstrates that explicit morpho-syntactic supervision can significantly enhance contextual disambiguation in lemmatization, and provides public code and models to support further research and application in NLP pipelines.

Abstract

English verbs have multiple forms. For instance, talk may also appear as talks, talked or talking, depending on the context. The NLP task of lemmatization seeks to map these diverse forms back to a canonical one, known as the lemma. We present a simple joint neural model for lemmatization and morphological tagging that achieves state-of-the-art results on 20 languages from the Universal Dependencies corpora. Our paper describes the model in addition to training and decoding procedures. Error analysis indicates that joint morphological tagging and lemmatization is especially helpful in low-resource lemmatization and languages that display a larger degree of morphological complexity. Code and pre-trained models are available at https://sigmorphon.github.io/sharedtasks/2019/task2/.

Figures (6)

• Figure 1: Our structured neural model shown as a hybrid (directed-undirected) graphical model koller2009probabilistic. Notionally, the $w_i$ denote inflected word forms, the $m_i$ denote morphological tags and the $\ell_i$ denote lemmata.
• Figure 2: Example of a morphologically tagged (in purple) and lemmatized (in red) sentence in Russian using the annotation scheme provided in the UD dataset. The translation is given below (in blue).
• Figure 3: We present performance (in accuracy) averaged over the 20 languages from UD we consider. Our method (second from the left) significantly outperforms the strongest baseline (fourth from the left; N18-1126). The blue column is a skyline that gives our model gold tags during decoding, showing improved tagging should lead to better lemmatization. The remaining are baselines described in \ref{['sec:baselines']}.
• Figure 4: Improvement on validation set with crunching over greedy decoding for different values of $k$.
• Figure 5: Dev accuracy breakdown by type of inflected form on all languages comparing our system with greedy decoding against our run of Lematus-ch20, colored by relative improvement in percentage. In each entry, the bottom score is from Lematus-ch20 and the top one is from our system, and the number in the parenthesis is the number of tokens for the corresponding setting.