Testing the Test: Observations When Assessing Visualization Literacy of Domain Experts

TL;DR

The paper investigates how time pressure and test design in Mini-VLAT affect visualization literacy assessment of domain experts. Through structured interviews across social science, architecture, engineering, and computer science, it identifies stress, ambiguities, and domain-relevance as key issues, and proposes design iterations (e.g., removing time limits, adding context, domain adaptation) to improve usability and validity. It outlines practical use cases for Mini-VLAT in design studies, crowd-sourced filtering, study interpretation, and expert training, while emphasizing ethical and cultural considerations. The work provides concrete guidelines for refining literacy assessments to better support domain-specific evaluations and invites future validation of domain-adaptive versions. Overall, it highlights that carefully framed, less stressful tests can yield more meaningful insights when evaluating visualization literacy in specialized populations.

Abstract

Various standardized tests exist that assess individuals' visualization literacy. Their use can help to draw conclusions from studies. However, it is not taken into account that the test itself can create a pressure situation where participants might fear being exposed and assessed negatively. This is especially problematic when testing domain experts in design studies. We conducted interviews with experts from different domains performing the Mini-VLAT test for visualization literacy to identify potential problems. Our participants reported that the time limit per question, ambiguities in the questions and visualizations, and missing steps in the test procedure mainly had an impact on their performance and content. We discuss possible changes to the test design to address these issues and how such assessment methods could be integrated into existing evaluation procedures.

Figures (6)

• Figure 1: Six questions from Mini-VLAT with which participants had difficulties. The issues in the visualizations and questions are highlighted in red. Visualizations reprinted with permission from Pandey and Ottley pandey_minivlat_2023 © John Wiley & Sons, Inc., and adapted.
• Figure 2: Mini-VLAT test results with 13 participants from the architecture domain across 12 questions: (a) The number of correct, incorrect, and missing answers per question. (b) The completion time per question. The whiskers represent 95% confidence intervals.
• Figure 3: The study procedure consisted of a questionnaire using LimeSurvey and a structured interview. All participants had to perform the Mini-VLAT test before the interview.
• Figure 4: Individual questions from Mini-VLAT were applied to the defined design ideas (D2--D5). They were shown during the interview and evaluated in comparison to Mini-VLAT. Visualizations reprinted with permission from Pandey and Ottley pandey_minivlat_2023 © John Wiley & Sons, Inc.
• Figure 5: Questionnaire results: (a) Participants' answers to the questions (Q1 to Q12) in the same order. Mistakes made by individual participants are marked as red. (b) Participants' subjective assessment of their visualization skills before and after the test. (c) Participants' perceived workload ratings using NASA-TLX after completing Mini-VLAT.