Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations

Tatiane Ornelas, Allysson Allex Araújo, Júlia Araújo, Marina Araújo, Bianca Trinkenreich, Marcos Kalinowski

TL;DR

The paper investigates how experienced software engineering researchers perceive the integration of large language models into thematic analysis, addressing opportunities, risks, and methodological implications. It adopts a reflective ISERN workshop with 25 researchers to explore LLM-assisted TA across five TA phases, using color-coded canvases to document insights. Key contributions include an empirical synthesis of perceived opportunities and risks, discussion of tensions between automation and interpretive depth, and practical recommendations for responsible human–AI collaboration in TA. The findings suggest that LLMs can augment qualitative analysis in SE when paired with prompting literacy, continuous human oversight, and transparent documentation, but cannot replace researcher judgment or contextual understanding.

Abstract

[Context] Large Language Models (LLMs) are increasingly used to assist qualitative research in Software Engineering (SE), yet the methodological implications of this usage remain underexplored. Their integration into interpretive processes such as thematic analysis raises fundamental questions about rigor, transparency, and researcher agency. [Objective] This study investigates how experienced SE researchers conceptualize the opportunities, risks, and methodological implications of integrating LLMs into thematic analysis. [Method] A reflective workshop with 25 ISERN researchers guided participants through structured discussions of LLM-assisted open coding, theme generation, and theme reviewing, using color-coded canvases to document perceived opportunities, limitations, and recommendations. [Results] Participants recognized potential efficiency and scalability gains, but highlighted risks related to bias, contextual loss, reproducibility, and the rapid evolution of LLMs. They also emphasized the need for prompting literacy and continuous human oversight. [Conclusion] Findings portray LLMs as tools that can support, but not substitute, interpretive analysis. The study contributes to ongoing community reflections on how LLMs can responsibly enhance qualitative research in SE.

LLM-Assisted Thematic Analysis: Opportunities, Limitations, and Recommendations

TL;DR

The paper investigates how experienced software engineering researchers perceive the integration of large language models into thematic analysis, addressing opportunities, risks, and methodological implications. It adopts a reflective ISERN workshop with 25 researchers to explore LLM-assisted TA across five TA phases, using color-coded canvases to document insights. Key contributions include an empirical synthesis of perceived opportunities and risks, discussion of tensions between automation and interpretive depth, and practical recommendations for responsible human–AI collaboration in TA. The findings suggest that LLMs can augment qualitative analysis in SE when paired with prompting literacy, continuous human oversight, and transparent documentation, but cannot replace researcher judgment or contextual understanding.

Abstract

[Context] Large Language Models (LLMs) are increasingly used to assist qualitative research in Software Engineering (SE), yet the methodological implications of this usage remain underexplored. Their integration into interpretive processes such as thematic analysis raises fundamental questions about rigor, transparency, and researcher agency. [Objective] This study investigates how experienced SE researchers conceptualize the opportunities, risks, and methodological implications of integrating LLMs into thematic analysis. [Method] A reflective workshop with 25 ISERN researchers guided participants through structured discussions of LLM-assisted open coding, theme generation, and theme reviewing, using color-coded canvases to document perceived opportunities, limitations, and recommendations. [Results] Participants recognized potential efficiency and scalability gains, but highlighted risks related to bias, contextual loss, reproducibility, and the rapid evolution of LLMs. They also emphasized the need for prompting literacy and continuous human oversight. [Conclusion] Findings portray LLMs as tools that can support, but not substitute, interpretive analysis. The study contributes to ongoing community reflections on how LLMs can responsibly enhance qualitative research in SE.

Paper Structure

This paper contains 31 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Towards LLM-Assisted Thematic Analysis
  4. Phase 1: Familiarization with the Data
  5. Phase 2: Open Coding and Human Validation
  6. Phase 3: Generation of Initial Themes and Human Validation
  7. Phase 4: Reviewing Themes
  8. Phase 5: Producing the Report
  9. Concrete Example
  10. Open Coding (LLM)
  11. Human Validation of Coding
  12. Generation of Initial Themes (LLM)
  13. Human Validation of Themes
  14. Reviewing Themes (LLM)
  15. Human Analysis and Final Validation
  16. ...and 16 more sections

Figures (2)

  • Figure 1: Overview of LLM-Assisted TA (adapted from BraunClarke2006UsingThematicAnalysis)
  • Figure 2: Canvases built in the workshop