XSTEM: An exemplar-based stemming algorithm

TL;DR

XSTEM tackles English stemming by addressing the limitations of dictionary-dependent and purely rule-based approaches, notably under-/over-stemming and out-of-vocabulary failures. It blends word-oriented lookup via a reverse-trie with exemplar-based rule generalization, enabling fast, scalable, multi-pass stemming that outputs real-word forms without requiring a lexicon. The system is highly configurable, distributing rules across multiple suffix classes and maintaining separate exception and proper-name resources, with about 1500 exemplar suffixes across 14 classes. An open-source Python reference implementation demonstrates strong precision and recall, offering practical utility for information retrieval and NLP tasks while mitigating prior stemmer shortcomings.

Abstract

Stemming is the process of reducing related words to a standard form by removing affixes from them. Existing algorithms vary with respect to their complexity, configurability, handling of unknown words, and ability to avoid under- and over-stemming. This paper presents a fast, simple, configurable, high-precision, high-recall stemming algorithm that combines the simplicity and performance of word-based lookup tables with the strong generalizability of rule-based methods to avert problems with out-of-vocabulary words.