ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment

TL;DR

The domain and language diversity in ReadMe++ enable us to test more effective few-shot prompting, and identify shortcomings in state-of-the-art unsupervised methods, and reveal exciting results of superior domain generalization and enhanced cross-lingual transfer capabilities by models trained on ReadMe++.

Abstract

We present a comprehensive evaluation of large language models for multilingual readability assessment. Existing evaluation resources lack domain and language diversity, limiting the ability for cross-domain and cross-lingual analyses. This paper introduces ReadMe++, a multilingual multi-domain dataset with human annotations of 9757 sentences in Arabic, English, French, Hindi, and Russian, collected from 112 different data sources. This benchmark will encourage research on developing robust multilingual readability assessment methods. Using ReadMe++, we benchmark multilingual and monolingual language models in the supervised, unsupervised, and few-shot prompting settings. The domain and language diversity in ReadMe++ enable us to test more effective few-shot prompting, and identify shortcomings in state-of-the-art unsupervised methods. Our experiments also reveal exciting results of superior domain generalization and enhanced cross-lingual transfer capabilities by models trained on ReadMe++. We will make our data publicly available and release a python package tool for multilingual sentence readability prediction using our trained models at: https://github.com/tareknaous/readme

Figures (19)

• Figure 1: Language distribution per each domain in ReadMe++. Example sentences from each language are shown along with their human-annotated readability levels on a 6-point scale (1: easiest, 6: hardest).
• Figure 2: Distribution of sentence lengths across readability levels in the English portion of ReadMe++, compared with CEFR-SP cefr. ReadMe++ offers a wider coverage of lengths and readability levels.
• Figure 3: Average readability rating and sentence length per domain in the English portion of ReadMe++. Domain diversity presents additional challenges for readability assessment. Certain domains may be within the same readability range (e.g. [2, 3] that corresponds to A2 and B1 levels) but have varying lengths, while sentences within a length range (e.g. [12, 17] tokens) could be spread across the whole readability spectrum.
• Figure 4: Pearson correlation ($\rho$) of fine-tuned multilingual and monolingual LMs, as well as prompted GPT3.5, GPT4, Aya23-8b, Llama2-7b, and Llama3.1-8b models with 5-shot examples, on the test set of ReadMe++. The small ($_S$), base ($_B$), and large ($_L$) sizes of the models are used. We report the min/max/average of performance across 5 runs using random seeds for fine-tuning initialization, or random sets of demonstrations in prompting.
• Figure 5: Effect of domain diversity of in-context examples on Llama2-7b performance on ReadMe++ (en). Correlation is greatly improved when examples are sampled from an increasing number of domains.