Measuring and predicting variation in the difficulty of questions about data visualizations

TL;DR

This study tackles why data-visualization tasks vary in difficulty by aggregating 230 items from five widely used literacy assessments and administering them under uniform testing conditions to 426 participants. Using logistic mixed-effects modeling, the authors show that item difficulty spans the full spectrum and is reliably estimated, with test type, graph type, and task type each contributing to performance but leaving substantial unexplained variance. The best-fitting model combines all three feature classes and their interaction, yet still falls far short of the estimated item-level reliability, indicating important item-specific factors beyond these features. The work advocates for unified, mechanistically grounded measures of data-visualization literacy to enable precise predictions, cross-test comparisons, and improved educational tools.

Abstract

Understanding what is communicated by data visualizations is a critical component of scientific literacy in the modern era. However, it remains unclear why some tasks involving data visualizations are more difficult than others. Here we administered a composite test composed of five widely used tests of data visualization literacy to a large sample of U.S. adults (N=503 participants).We found that items in the composite test spanned the full range of possible difficulty levels, and that our estimates of item-level difficulty were highly reliable. However, the type of data visualization shown and the type of task involved only explained a modest amount of variation in performance across items, relative to the reliability of the estimates we obtained. These results highlight the need for finer-grained ways of characterizing these items that predict the reliable variation in difficulty measured in this study, and that generalize to other tests of data visualization understanding.

Figures (6)

• Figure 1: We used 230 items from five popular tests of data visualization literacy, which vary in graph type and task type.
• Figure 2: Our item sampling procedure selected 46 items from the total set of 230 items for each participant (left). Everyone was presented with multiple-choice items from all six tests, with a 60 second time limit to answer each question (right).
• Figure 3: Average performance across all items. Items belonging to the same test share the same color. Error bars represent bootstrapped 95% confidence intervals.
• Figure 4: Performance across different tests (A), task types (B), and graph types (C), measured by the mean proportion of correct responses. Opaque dots indicate the mean proportion of correct responses for individual items. Error bars represent bootstrapped 95% confidence intervals.
• Figure 5: Comparison of model fit across mixed-effects logistic regression models, measured using marginal $R^2$. Green circles indicate fixed effects included in the model, with '*' indicating an interaction between fixed effects. The noise ceiling is estimated by computing the squared Pearson correlation between split halves over participants' data. Gray bands represent the expected variation $R^2$ due to sampling variability across samples of participants, estimated by bootstrap resampling.