Chasing Meaning and/or Insight? A Survey on Evaluation Practices at the Intersection of Visualization and the Humanities

TL;DR

The paper addresses how to evaluate visualizations at the VIS*H intersection, where analytic utility must harmonize with interpretive meaning. It conducts a large-scale, mixed-methods survey of 171 VIS*H design studies to map evaluation practices, rigor, and workflows, using a two-stage categorization and an ordinal logistic model to link methods to quality. Key findings show that monomethod evaluations are common and associate with lower rigor, while multi-method workflows that include log analysis, structured questionnaires, and interviews predict higher quality; simply increasing the number of methods yields little benefit unless the methods are well-composed. The authors argue for a paradigm shift toward grounding visualizations in provenance, humanities theories, and interpretive criteria to bridge discovery and discursive practices, and they provide concrete recommendations and a public dataset to support ongoing methodological development in VIS*H evaluation.

Abstract

The intersection of visualization and the humanities (VIS*H) is marked by a tension between chasing analytical "insight" and interpretive "meaning." The effectiveness of visualization techniques hinges on established evaluation frameworks that assess both analytical utility and communicative efficacy, creating a potential mismatch with the non-positivist, interpretive aims of humanities scholarship. To examine how this tension manifests in practice, we systematically surveyed 171 VIS*H design studies to analyze their evaluation workflows and rigor according to standard practice. Our findings reveal recurring flaws, such as an over-reliance on monomethod approaches, and show that higher-quality evaluations emerge from workflows that effectively triangulate diverse evidence. From these findings, we derive recommendations to refine quality and validation criteria for humanities visualizations, and juxtapose them to ongoing critical debates in the field, ultimately arguing for a paradigm shift that can reconcile the advantages of established validation techniques with the interpretive depth required for humanistic inquiry.

Figures (5)

• Figure 1: PRISMA flow diagram detailing the literature selection process. Starting from a seed corpus of seminal works, 1,317 records were identified via forward and backward snowballing. The multistage screening process, which filtered for sufficient evaluation detail and relevance to humanities inquiry, resulted in a final corpus of 171 design studies included in the survey.
• Figure 2: Descriptive overview of the 171 surveyed papers across key dimensions. The data highlights a predominance of Literature and History applications (top left) and a strong focus on Expert users (bottom left). Notably, the distribution of Assessed Rigor Scores (bottom right) is right-skewed, indicating that while low-to-moderate rigor evaluations are common, high-rigor studies remain a minority practice. The * denotes categories where multiple tags can apply.
• Figure 3: Co-occurrence matrix of evaluation methods found in our sample. Each cell shows the frequency with which two methods were used together in the same study, with brighter colors indicating higher absolute frequencies. Notable patterns include the frequent co-use of interviews with testing (34) and with questionnaires/surveys (21), as well as the predominant use of case studies as standalone methods (50).
• Figure 4: Overview heatmap of the various evaluation methodologies employed across the eight distinct clusters. The x-axis enumerates the nine evaluation methods incorporated in our analysis, while the y-axis represents the cluster number (1-8). Each cell depicts the proportion of papers within each cluster that utilized a specific evaluation method.
• Figure 5: Cluster patterns for different categories: qualitative vs. quantitative methods, summative vs. formative approach, no. (Q1-Q3 range, Md) and kind of study participants, and visualization characteristics (multiple views, uncertainty awareness, availability).