Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research

TL;DR

This study addresses the challenge of conducting rigorous qualitative analysis in large-scale, multi-site health-services research by introducing a model- and task-agnostic framework for human–LLM collaboration. The authors demonstrate two LLM-assisted tasks within the iPATH diabetes-care program: (i) generating comparative, domain-focused feedback for Federally Qualified Health Centers and (ii) deductive coding to refine a practice-transformation intervention, using retrieval-augmented generation and embedding-based retrieval to manage large transcript datasets. The results show that LLMs can organize data and accelerate analysis while researchers retain interpretive control and grounding in the data, though careful validation is required to preserve depth and context. The work provides actionable guidance for integrating LLMs into qualitative health-services research, highlighting both efficiency gains and the need for rigorous oversight to maintain rigor, transparency, and data integrity, with implications for scaling intervention design and practice improvement in diabetes care at FQHCs.

Abstract

Large language models (LLMs) show promise for improving the efficiency of qualitative analysis in large, multi-site health-services research. Yet methodological guidance for LLM integration into qualitative analysis and evidence of their impact on real-world research methods and outcomes remain limited. We developed a model- and task-agnostic framework for designing human-LLM qualitative analysis methods to support diverse analytic aims. Within a multi-site study of diabetes care at Federally Qualified Health Centers (FQHCs), we leveraged the framework to implement human-LLM methods for (1) qualitative synthesis of researcher-generated summaries to produce comparative feedback reports and (2) deductive coding of 167 interview transcripts to refine a practice-transformation intervention. LLM assistance enabled timely feedback to practitioners and the incorporation of large-scale qualitative data to inform theory and practice changes. This work demonstrates how LLMs can be integrated into applied health-services research to enhance efficiency while preserving rigor, offering guidance for continued innovation with LLMs in qualitative research.

Figures (6)

• Figure 1: The framework we developed and applied for developing task-specific human-LLM qualitative analysis methods.
• Figure 2: Illustrative differences in cross-site synthesis output by human and LLM (independently) for telehealth and appointment management themes within the "Information and Communication Technology" domain
• Figure 3: Final Task 1 human-LLM method. Step 1, qualitative researchers define domains and create site-level summaries for each one. Step 2, LLM groups data for each domain into themes and provides cross-site synthesis. Step 3, qualitative researcher modifies LLM thematic groups. Step 4, qualitative researchers finalize cross-site synthesis for each domain, highlighting actionable insights and best practices.
• Figure 4: The human-LLM coding method for each code. Researchers first generate sub-questions for relevant aspects of the code, then discuss and refine them as a team to ensure alignment. For each sub-question, researchers add two additional questions: 1) if there are examples in the sub-question, the same question without examples (‘example’ bias), and 2) a question on the same topic focused on challenges and barriers (‘positivity’ bias). Next, for each sub-question, we perform Retrieval Augmented Generation: embedding-based retrieval identifies relevant excerpts, and the LLM is prompted to answer using these excerpts. An automated script concatenates LLM outputs for all questions into a single output, and merges duplicate bullet points tied to the same quote, and validates that all quotes appear in the interview text. Finally, the LLM sorts the validated bullet points, which are then provided to the research team.
• Figure 5: Different summary statement/quote points brought up by the human and LLMs for the ‘patient–provider relationship’ code. Human focuses on interpersonal dynamics and LLM on structures and processes. All other summary statement/quote points for this code were very similar.