Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasets

TL;DR

The paper addresses annotation quality in multi-dialect Arabic datasets by leveraging ALDi, a continuous measure of dialect divergence on $[0,1]$, to guide annotator routing. It analyzes 15 public datasets across six sentence-classification tasks, computing per-sample ALDi with Sentence-ALDi, binning into 10 intervals, and assessing interannotator agreement via % full agree alongside Pearson's $\rho$ and slope $m$, with logistic regression as a robustness check. Results show a strong negative relation between ALDi and agreement for non-DI tasks, while DI tasks exhibit higher agreement at high ALDi due to clearer dialect cues; findings vary by label type and dataset. The study advocates a practical routing strategy: automatically estimate ALDi, perform DI on high-ALDi samples, and route them to native speakers of the corresponding dialect to improve data quality and annotation efficiency in multi-dialect Arabic NLP.

Abstract

On annotating multi-dialect Arabic datasets, it is common to randomly assign the samples across a pool of native Arabic speakers. Recent analyses recommended routing dialectal samples to native speakers of their respective dialects to build higher-quality datasets. However, automatically identifying the dialect of samples is hard. Moreover, the pool of annotators who are native speakers of specific Arabic dialects might be scarce. Arabic Level of Dialectness (ALDi) was recently introduced as a quantitative variable that measures how sentences diverge from Standard Arabic. On randomly assigning samples to annotators, we hypothesize that samples of higher ALDi scores are harder to label especially if they are written in dialects that the annotators do not speak. We test this by analyzing the relation between ALDi scores and the annotators' agreement, on 15 public datasets having raw individual sample annotations for various sentence-classification tasks. We find strong evidence supporting our hypothesis for 11 of them. Consequently, we recommend prioritizing routing samples of high ALDi scores to native speakers of each sample's dialect, for which the dialect could be automatically identified at higher accuracies.

Figures (7)

• Figure 1: Scatter plots showing the relationship between binned ALDi scores (x-axis) and the percentage of samples with full annotator agreement (y-axis). The histogram represents the # of samples per bin (with min and max values for any bin labeled on the right-hand axis). The slope of the best-fitting line ($m$) is shown, and to enable visual comparison of slopes, all plots have the same y-axis scale (possibly shifted up or down). Note: Statistically significant (p<0.05) correlation coefficients ($\rho$) are marked with *.
• Figure C1: For each dataset, plots show the estimated probability of full agreement according to each dataset's fitted logistic regression model. Under each plot, the coefficient of ALDi with its 95% confidence interval is visualized. Nearly all datasets (marked with *) have confidence intervals that do not include zero, meaning the effect of ALDi is statistically significant at $p< 0.05$. Negative coefficients indicate that higher ALDi scores predict lower agreement.
• Figure D2: The trends for the classes of the Saracasm Detection datasets. Statistically significant correlation coefficients ($\rho$) are marked with *.
• Figure D3: The trends for the classes of the Sentiment Analysis datasets. Statistically significant correlation coefficients ($\rho$) are marked with *.
• Figure D4: The trends for the classes of the Offensive Text Classification and Hate Speech datasets. Statistically significant correlation coefficients ($\rho$) are marked with *.