Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study

TL;DR

This paper empirically investigates whether large language model agents can align with human ratings in visualization studies. Through three studies, it shows that model-based agents can simulate human reasoning and ratings for basic tasks but struggle to capture user diversity, and that alignment with human data is strongest when expert pre-experiment confidence is high. The work further explores prompt design, visual/textual cue influence, and knowledge injection (RAG) as strategies to improve alignment, while highlighting robustness issues and potential biases. The findings suggest that agent-based simulations can support fast prototyping and low-cost exploration of experimental designs, but cannot replace real user studies. Overall, the research outlines practical pathways and cautions for integrating LLM agents into visualization evaluation pipelines.

Abstract

Large language models encode knowledge in various domains and demonstrate the ability to understand visualizations. They may also capture visualization design knowledge and potentially help reduce the cost of formative studies. However, it remains a question whether large language models are capable of predicting human feedback on visualizations. To investigate this question, we conducted three studies to examine whether large model-based agents can simulate human ratings in visualization tasks. The first study, replicating a published study involving human subjects, shows agents are promising in conducting human-like reasoning and rating, and its result guides the subsequent experimental design. The second study repeated six human-subject studies reported in literature on subjective ratings, but replacing human participants with agents. Consulting with five human experts, this study demonstrates that the alignment of agent ratings with human ratings positively correlates with the confidence levels of the experts before the experiments. The third study tests commonly used techniques for enhancing agents, including preprocessing visual and textual inputs, and knowledge injection. The results reveal the issues of these techniques in robustness and potential induction of biases. The three studies indicate that language model-based agents can potentially simulate human ratings in visualization experiments, provided that they are guided by high-confidence hypotheses from expert evaluators. Additionally, we demonstrate the usage scenario of swiftly evaluating prototypes with agents. We discuss insights and future directions for evaluating and improving the alignment of agent ratings with human ratings. We note that simulation may only serve as complements and cannot replace user studies.

Figures (8)

• Figure 1: Agent ratings for time series visualization tasks.
• Figure 2: The data preparation steps. The values represent the number of remaining papers after each step. For VIS papers, we skip the keyword matching step and utilized vispubdata 7583708 dataset to obtain all paper titles.
• Figure 3: Results of agent ratings for fit estimations under various noise levels (top) and decentering (bottom). Error bars represent 95% bootstrap CIs.
• Figure 4: Proportion of agent ratings $\geq$ 3 for imputation methods addressing uncertainty. They feature two tasks (estimating the average and trend) across three conditions: (A) text and images combined, (B) text only with the same prompt, and (C) text only with a general description.
• Figure 5: RAG process designed for knowledge injection in the experiment of Time Series in Study III.