Beyond Literacy: Predicting Interpretation Correctness of Visualizations with User Traits, Item Difficulty, and Rasch Scores
Davide Falessi, Silvia Golia, Angela Locoro
TL;DR
The paper tackles the problem of predicting whether a given person will interpret a data visualization correctly before exposure to the item (P-HIC). It operationalizes P-HIC as a binary prediction task using 22 pre-interaction features spanning Human Profile, Human Performance, and Item difficulty, and evaluates three ML models with/without feature selection on data from 1,083 participants answering 32 items. Logistic Regression with feature selection emerges as the strongest approach, achieving a median $AUC$ of approximately 0.72 and a median $\kappa$ of about 0.32, with RaschDifficulty consistently the most informative predictor. The findings support the feasibility of adaptive and personalized data-visualization literacy assessments, enabling runtime item selection and targeted training to improve interpretation accuracy while reducing cognitive burden. The study also highlights the value of psychometric features and performance history in understanding and predicting interpretation outcomes, with implications for cross-cultural validity and equitable deployment.
Abstract
Data Visualization Literacy assessments are typically administered via fixed sets of Data Visualization items, despite substantial heterogeneity in how different people interpret the same visualization. This paper presents and evaluates an approach for predicting Human Interpretation Correctness (P-HIC) of data visualizations; i.e., anticipating whether a specific person will interpret a data visualization correctly or not, before exposure to that DV, enabling more personalized assessment and training. We operationalize P-HIC as a binary classification problem using 22 features spanning Human Profile, Human Performance, and Item difficulty (including ExpertDifficulty and RaschDifficulty). We evaluate three machine-learning models (Logistic Regression model, Random Forest, Multi Layer Perceptron) with and without feature selection, using a survey with 1,083 participants who answered 32 Data Visualization items (eight data visualizations per four items), yielding 34,656 item responses. Performance is assessed via a ten-time ten-fold cross-validation in each 32 (item-specific) datasets, using AUC and Cohen's kappa. Logistic Regression model with feature selection is the best-performing approach, reaching a median AUC of 0.72 and a median kappa of 0.32. Feature analyses show RaschDifficulty as the dominant predictor, followed by experts' ratings and prior correctness (PercCorrect), whose relevance increases across sessions. Profile information did not particularly support P-HIC. Our results support the feasibility of anticipating misinterpretations of data visualizations, and motivate the runtime selection of data visualizations items tailored to an audience, thereby improving the efficiency of Data Visualization Literacy assessment and targeted training.