Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis

He Zhang, Chuhao Wu, Jingyi Xie, Yao Lyu, Jie Cai, John M. Carroll

TL;DR

The study tackles improving qualitative thematic analysis with ChatGPT by developing a human-centric prompt-design framework. Through a two-stage Empirical investigation (pilot and formal) with 17 participants and collaboration with 13 qualitative researchers, the authors identify barriers to AI-assisted analysis and derive a structured prompts framework describing task background, methodology, data formats, and output formats to boost robustness. Key findings show that increasing transparency and guiding prompts enhances trust and enables more interpretable, traceable outputs, shifting researchers' attitudes from skepticism to acceptance. The work highlights ethical considerations and outlines future directions, including an integrated LLM toolkit and roles for AI as tool or co-researcher, with implications for democratizing access to qualitative analysis and improving efficiency in large-scale data coding.

Abstract

AI tools, particularly large-scale language model (LLM) based applications such as ChatGPT, have the potential to simplify qualitative research. Through semi-structured interviews with seventeen participants, we identified challenges and concerns in integrating ChatGPT into the qualitative analysis process. Collaborating with thirteen qualitative researchers, we developed a framework for designing prompts to enhance the effectiveness of ChatGPT in thematic analysis. Our findings indicate that improving transparency, providing guidance on prompts, and strengthening users' understanding of LLMs' capabilities significantly enhance the users' ability to interact with ChatGPT. We also discovered and revealed the reasons behind researchers' shift in attitude towards ChatGPT from negative to positive. This research not only highlights the importance of well-designed prompts in LLM applications but also offers reflections for qualitative researchers on the perception of AI's role. Finally, we emphasize the potential ethical risks and the impact of constructing AI ethical expectations by researchers, particularly those who are novices, on future research and AI development.

Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis

TL;DR

The study tackles improving qualitative thematic analysis with ChatGPT by developing a human-centric prompt-design framework. Through a two-stage Empirical investigation (pilot and formal) with 17 participants and collaboration with 13 qualitative researchers, the authors identify barriers to AI-assisted analysis and derive a structured prompts framework describing task background, methodology, data formats, and output formats to boost robustness. Key findings show that increasing transparency and guiding prompts enhances trust and enables more interpretable, traceable outputs, shifting researchers' attitudes from skepticism to acceptance. The work highlights ethical considerations and outlines future directions, including an integrated LLM toolkit and roles for AI as tool or co-researcher, with implications for democratizing access to qualitative analysis and improving efficiency in large-scale data coding.

Abstract

AI tools, particularly large-scale language model (LLM) based applications such as ChatGPT, have the potential to simplify qualitative research. Through semi-structured interviews with seventeen participants, we identified challenges and concerns in integrating ChatGPT into the qualitative analysis process. Collaborating with thirteen qualitative researchers, we developed a framework for designing prompts to enhance the effectiveness of ChatGPT in thematic analysis. Our findings indicate that improving transparency, providing guidance on prompts, and strengthening users' understanding of LLMs' capabilities significantly enhance the users' ability to interact with ChatGPT. We also discovered and revealed the reasons behind researchers' shift in attitude towards ChatGPT from negative to positive. This research not only highlights the importance of well-designed prompts in LLM applications but also offers reflections for qualitative researchers on the perception of AI's role. Finally, we emphasize the potential ethical risks and the impact of constructing AI ethical expectations by researchers, particularly those who are novices, on future research and AI development.
Paper Structure (42 sections, 5 figures, 3 tables)

This paper contains 42 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. ChatGPT and Prompt Engineering
  4. Current Methodologies and Features of Thematic Analysis
  5. AI-augmented Qualitative Analysis
  6. Methods
  7. Data Collection
  8. Pilot Study
  9. Formal Study
  10. Participants Recruitment
  11. Data Analysis
  12. Users' Experiences and Challenges with ChatGPT
  13. Lack of Transparency
  14. Difficulty of Designing Propmpts
  15. Insufficient Understanding of ChatGPT's Capabilities
  16. ...and 27 more sections

Figures (5)

  • Figure 1: An example of some results: Participant (P15) added requirements regarding the format of the output results in the prompts. The original prompt text reads: "For the output, put the result in a table. The first column is the name of the theme, the second column is its frequency, the third column includes the quotes that belong to this theme and the name of the participant who made this comment, one row per quote.".
  • Figure 2: An example of how to transfer the table from ChatGPT to Excel.
  • Figure 3: Examples of ChatGPT's output after adding priority requirements.
  • Figure 4: Examples of ChatGPT's output from P8's prompts.
  • Figure 5: Some of the results obtained by participants using ChatGPT. On the left is the result for P5, and on the right is the result for P6.