Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-TA: An LLM-Enhanced Thematic Analysis Pipeline for Transcripts from Parents of Children with Congenital Heart Disease

Muhammad Zain Raza, Jiawei Xu, Terence Lim, Lily Boddy, Carlos M. Mery, Andrew Well, Ying Ding

TL;DR

The paper tackles the labor-intensive nature of inductive thematic analysis (TA) in healthcare by introducing LLM-TA, an LLM-augmented TA pipeline that analyzes lengthy, de-identified transcripts from parents of children with AAOCA. It implements a two-stage process—Stage 1 for granular code generation on chunked transcripts and Stage 2 for theme synthesis—using chunking, LangChain, and prompting strategies (zero-shot, one-shot, Reflexion). Evaluated against human-ground-truth themes, LLM-TA shows significant efficiency gains (roughly 97% time reduction) and improvements in thematic similarity and usefulness compared with baselines, though domain-expert collaboration remains essential due to representativeness and clinical-context limitations. The study highlights the importance of expert involvement to guide prompts and interpretation in high-stakes TA, and outlines concrete directions for improving reliability and generalizability across conditions.

Abstract

Thematic Analysis (TA) is a fundamental method in healthcare research for analyzing transcript data, but it is resource-intensive and difficult to scale for large, complex datasets. This study investigates the potential of large language models (LLMs) to augment the inductive TA process in high-stakes healthcare settings. Focusing on interview transcripts from parents of children with Anomalous Aortic Origin of a Coronary Artery (AAOCA), a rare congenital heart disease, we propose an LLM-Enhanced Thematic Analysis (LLM-TA) pipeline. Our pipeline integrates an affordable state-of-the-art LLM (GPT-4o mini), LangChain, and prompt engineering with chunking techniques to analyze nine detailed transcripts following the inductive TA framework. We evaluate the LLM-generated themes against human-generated results using thematic similarity metrics, LLM-assisted assessments, and expert reviews. Results demonstrate that our pipeline outperforms existing LLM-assisted TA methods significantly. While the pipeline alone has not yet reached human-level quality in inductive TA, it shows great potential to improve scalability, efficiency, and accuracy while reducing analyst workload when working collaboratively with domain experts. We provide practical recommendations for incorporating LLMs into high-stakes TA workflows and emphasize the importance of close collaboration with domain experts to address challenges related to real-world applicability and dataset complexity. https://github.com/jiaweixu98/LLM-TA

LLM-TA: An LLM-Enhanced Thematic Analysis Pipeline for Transcripts from Parents of Children with Congenital Heart Disease

TL;DR

The paper tackles the labor-intensive nature of inductive thematic analysis (TA) in healthcare by introducing LLM-TA, an LLM-augmented TA pipeline that analyzes lengthy, de-identified transcripts from parents of children with AAOCA. It implements a two-stage process—Stage 1 for granular code generation on chunked transcripts and Stage 2 for theme synthesis—using chunking, LangChain, and prompting strategies (zero-shot, one-shot, Reflexion). Evaluated against human-ground-truth themes, LLM-TA shows significant efficiency gains (roughly 97% time reduction) and improvements in thematic similarity and usefulness compared with baselines, though domain-expert collaboration remains essential due to representativeness and clinical-context limitations. The study highlights the importance of expert involvement to guide prompts and interpretation in high-stakes TA, and outlines concrete directions for improving reliability and generalizability across conditions.

Abstract

Thematic Analysis (TA) is a fundamental method in healthcare research for analyzing transcript data, but it is resource-intensive and difficult to scale for large, complex datasets. This study investigates the potential of large language models (LLMs) to augment the inductive TA process in high-stakes healthcare settings. Focusing on interview transcripts from parents of children with Anomalous Aortic Origin of a Coronary Artery (AAOCA), a rare congenital heart disease, we propose an LLM-Enhanced Thematic Analysis (LLM-TA) pipeline. Our pipeline integrates an affordable state-of-the-art LLM (GPT-4o mini), LangChain, and prompt engineering with chunking techniques to analyze nine detailed transcripts following the inductive TA framework. We evaluate the LLM-generated themes against human-generated results using thematic similarity metrics, LLM-assisted assessments, and expert reviews. Results demonstrate that our pipeline outperforms existing LLM-assisted TA methods significantly. While the pipeline alone has not yet reached human-level quality in inductive TA, it shows great potential to improve scalability, efficiency, and accuracy while reducing analyst workload when working collaboratively with domain experts. We provide practical recommendations for incorporating LLMs into high-stakes TA workflows and emphasize the importance of close collaboration with domain experts to address challenges related to real-world applicability and dataset complexity. https://github.com/jiaweixu98/LLM-TA

Paper Structure

This paper contains 10 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Stage 1: Initial code generation on chunked transcripts.
  4. Stage 2: Theme Generation.
  5. Prompting Strategies.
  6. Experiment and Evaluation
  7. AAOCA Patients Focus Group Transcript Dataset.
  8. Results
  9. Conclusion
  10. Acknowledgements

Figures (3)

  • Figure 1: Pipeline for LLM-enhanced thematic analysis (LLM-TA) of transcripts from parents of children with AAOCA.
  • Figure 2: Prompting strategies in the LLM-TA code generation stage. These strategies are also used for theme generation.
  • Figure 3: (a) Similarity heatmap of themes generated by the LLM-TA (zero-shot) method compared with human themes using all-MiniLM-L6-v2. (b) Similarity heatmap of themes generated by the LLM-TA (zero-shot) method compared with human themes using all-MiniLM-L6-v2, with a threshold score of 0.56.