Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features
Bruce W. Lee, Yoo Sung Jang, Jason Hyung-Jong Lee
TL;DR
This work advances readability assessment by introducing advanced semantic features derived from topic distributions and by building a large 255-feature LingFeat toolkit to systematically capture linguistic cues. It then demonstrates that a simple hybrid architecture—combining neural model soft labels with handcrafted features fed into a non-neural learner—can achieve state-of-the-art results on standard RA datasets, including near-perfect accuracy on OneStopEnglish. The approach shows particular strength on small datasets, improves robustness via diverse feature signals, and provides insights into when and why hybrid models outperform purely neural or purely handcrafted approaches. Overall, the combination of deep semantics and hybrid modeling yields practical, data-efficient gains in readability prediction with strong cross-dataset potential.
Abstract
We report two essential improvements in readability assessment: 1. three novel features in advanced semantics and 2. the timely evidence that traditional ML models (e.g. Random Forest, using handcrafted features) can combine with transformers (e.g. RoBERTa) to augment model performance. First, we explore suitable transformers and traditional ML models. Then, we extract 255 handcrafted linguistic features using self-developed extraction software. Finally, we assemble those to create several hybrid models, achieving state-of-the-art (SOTA) accuracy on popular datasets in readability assessment. The use of handcrafted features help model performance on smaller datasets. Notably, our RoBERTA-RF-T1 hybrid achieves the near-perfect classification accuracy of 99%, a 20.3% increase from the previous SOTA.