Bridging Quantitative and Qualitative Methods for Visualization Research: A Data/Semantics Perspective in Light of Advanced AI

TL;DR

The paper tackles how to bridge quantitative and qualitative evaluation methods in visualization amid advances in AI. It introduces a process model that iteratively enriches data with semantics, enabling integrated analysis of multimodal user study data. The model is motivated by eye-tracking studies and coding of data-rich observations, and discusses the role of AI, human roles, and potential open research directions. The work highlights open issues around reliability, ethics, and evaluation, and calls the visualization community to adopt flexible, semantics-aware methodologies.

Abstract

This paper revisits the role of quantitative and qualitative methods in visualization research in the context of advancements in artificial intelligence (AI). The focus is on how we can bridge between the different methods in an integrated process of analyzing user study data. To this end, a process model of - potentially iterated - semantic enrichment and transformation of data is proposed. This joint perspective of data and semantics facilitates the integration of quantitative and qualitative methods. The model is motivated by examples of own prior work, especially in the area of eye tracking user studies and coding data-rich observations. Finally, there is a discussion of open issues and research opportunities in the interplay between AI, human analyst, and qualitative and quantitative methods for visualization research.

Figures (5)

• Figure 1: Heatmap of the distribution of attention (here, for a user study on visual search in maps). Image © 2017 IEEE. Reprinted with permission from Netzel el al. Netzel:2017:EvaluationVisualSearch.
• Figure 2: Eye tracking study data with enriched semantics: (top) Annotation by AOIs provides meaning to glanced-at regions of the stimulus, along with quantitative data about their relative frequency over time. (bottom/center-right) Scarf plots with the same color coding as for the AOIs above show the distribution of attention for each individual participant over time. (bottom/left) Dendrogram of hierarchical clustering of scanpaths for the participants. Image © 2014 ACM, reused from Kurzhals et al. Kurzhals:2014:ISeeCube.
• Figure 3: Interactive visualization system for trust building in AI-assisted labeling of video data associated with eye tracking recordings: (left) glyph-based visualization of prediction uncertainty of the ML classifier, (center) 2D embedding of image thumbnails around gaze positions, along with zoomed-in insets showing the interactive exploration of fixation details for incoming and outgoing connections, (right) panel with application settings and information. Image reprinted from © 2024 Koch et al. Koch:2024:ActiveGazeLabeling, licensed under CC BY 4.0.
• Figure 4: Visual interface for coding of data-rich user behavior, including transcribed text, several facets of eye tracking data (point-based data and AOIs), and interaction logging. Image © 2016 IEEE. Reprinted with permission from Blascheck el al. Blascheck:2016:VisualAnalysisCoding.
• Figure 5: Schematic process of research question, study design and execution, and iterative analysis of (possibly multimodal) study data. The key part is the analysis loop that keeps on transforming and enriching data with additional semantics to derive new data representations. Through the process, information is obtained at higher and higher levels of understanding. The analysis loop may consist of AI-based processing, user intervention, or a combination thereof.