When Qualitative Research Meets Large Language Model: Exploring the Potential of QualiGPT as a Tool for Qualitative Coding
He Zhang, Chuhao Wu, Jingyi Xie, Fiona Rubino, Sydney Graver, ChanMin Kim, John M. Carroll, Jie Cai
TL;DR
The paper addresses the bottlenecks of qualitative coding by introducing QualiGPT, an API-based tool designed to enhance efficiency, transparency, and accessibility in thematic analysis. It presents a design that combines prompt engineering, controlled data handling, and structured outputs, and validates the approach through three case studies involving inductive and deductive coding tasks with inter-rater reliability assessments. Results indicate that QualiGPT can substantially reduce analysis time, achieve meaningful agreement with human coders, and offer a more usable pipeline than traditional CAQDA or web-based ChatGPT approaches. The work highlights both the practical benefits of AI-assisted qualitative analysis and the need for careful human-in-the-loop supervision, data privacy considerations, and ongoing model evaluation as part of an evolving collaborative research paradigm.
Abstract
Qualitative research, renowned for its in-depth exploration of complex phenomena, often involves time-intensive analysis, particularly during the coding stage. Existing software for qualitative evaluation frequently lacks automatic coding capabilities, user-friendliness, and cost-effectiveness. The advent of Large Language Models (LLMs) like GPT-3 and its successors marks a transformative era for enhancing qualitative analysis. This paper introduces QualiGPT, a tool developed to address the challenges associated with using ChatGPT for qualitative analysis. Through a comparative analysis of traditional manual coding and QualiGPT's performance on both simulated and real datasets, incorporating both inductive and deductive coding approaches, we demonstrate that QualiGPT significantly improves the qualitative analysis process. Our findings show that QualiGPT enhances efficiency, transparency, and accessibility in qualitative coding. The tool's performance was evaluated using inter-rater reliability (IRR) measures, with results indicating substantial agreement between human coders and QualiGPT in various coding scenarios. In addition, we also discuss the implications of integrating AI into qualitative research workflows and outline future directions for enhancing human-AI collaboration in this field.