Expanding Horizons in HCI Research Through LLM-Driven Qualitative Analysis

TL;DR

Qualitative analysis in HCI is labor-intensive and difficult to scale. The authors propose an LLM-driven qualitative analysis framework evaluated with SBART cosine similarity, applying it to a dataset derived from three CHI/HCI papers and their open-ended questionnaire data. The work delivers two main contributions: (i) characterization of LLM-driven qualitative analysis and (ii) the first benchmark for evaluating LLMs in qualitative analysis, enabling reproducibility and scalability. This approach offers a pathway to more transparent, scalable qualitative research in HCI and opens opportunities for refining evaluation methods and expanding datasets for LLM-assisted analysis.

Abstract

How would research be like if we still needed to "send" papers typed with a typewriter? Our life and research environment have continually evolved, often accompanied by controversial opinions about new methodologies. In this paper, we embrace this change by introducing a new approach to qualitative analysis in HCI using Large Language Models (LLMs). We detail a method that uses LLMs for qualitative data analysis and present a quantitative framework using SBART cosine similarity for performance evaluation. Our findings indicate that LLMs not only match the efficacy of traditional analysis methods but also offer unique insights. Through a novel dataset and benchmark, we explore LLMs' characteristics in HCI research, suggesting potential avenues for further exploration and application in the field.

Figures (2)

• Figure 1: Three prompts used in the qualitative analysis LLMs system and the evaluation system.
• Figure 2: Cosine similarity results from SBERT. (A) violin plotted, (B) showing the relationship between the paper and the data. ID 1-27 10.1145/3313831.3376267, 28 10.1145/3290605.3300732, 29-32 10.1145/3290605.3300310