Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt Engineering for Large Language Model-assisted Inductive Thematic Analysis

Muhammad Talal Khalid, Ann-Perry Witmer

TL;DR

This paper addresses the need for methodological rigor in using large language models to support inductive thematic analysis by conducting a systematic literature review and proposing a structured four-step prompt engineering process. It maps current PE practices, identifies common strategies such as single- versus multi-prompt approaches, and discusses advanced techniques like decomposition, ensembling, and self-generated prompts to improve reliability and transparency. The authors emphasize that LLMs should augment rather than replace human researchers, and they outline ethical considerations and future research directions to establish replicable PE methods in ITA. Overall, the work provides a practical framework and guidelines to enhance the rigor, efficiency, and trustworthiness of LLM-assisted ITA research.

Abstract

The potential of large language models (LLMs) to mitigate the time- and cost- related challenges associated with inductive thematic analysis (ITA) has been extensively explored in the literature. However, the use of LLMs to support ITA has often been opportunistic, relying on ad hoc prompt engineering (PE) approaches, thereby undermining the reliability, transparency, and replicability of the analysis. The goal of this study is to develop a structured approach to PE in LLM-assisted ITA. To this end, a comprehensive review of the existing literature is conducted to examine how ITA researchers integrate LLMs into their workflows and, in particular, how PE is utilized to support the analytical process. Built on the insights generated from this review, four key steps for effective PE in LLM-assisted ITA are identified and extensively outlined. Furthermore, the study explores state-of-the-art PE techniques that can enhance the execution of these steps, providing ITA researchers with practical strategies to improve their analyses. In conclusion, the main contributions of this paper include: (i) it maps the existing research on LLM-assisted ITA to enable a better understanding of the rapidly developing field, (ii) it outlines a structured four-step PE process to enhance methodological rigor, (iii) it discusses the application of advanced PE techniques to support the execution of these steps, and (iv) it highlights key directions for future research.

Prompt Engineering for Large Language Model-assisted Inductive Thematic Analysis

TL;DR

This paper addresses the need for methodological rigor in using large language models to support inductive thematic analysis by conducting a systematic literature review and proposing a structured four-step prompt engineering process. It maps current PE practices, identifies common strategies such as single- versus multi-prompt approaches, and discusses advanced techniques like decomposition, ensembling, and self-generated prompts to improve reliability and transparency. The authors emphasize that LLMs should augment rather than replace human researchers, and they outline ethical considerations and future research directions to establish replicable PE methods in ITA. Overall, the work provides a practical framework and guidelines to enhance the rigor, efficiency, and trustworthiness of LLM-assisted ITA research.

Abstract

The potential of large language models (LLMs) to mitigate the time- and cost- related challenges associated with inductive thematic analysis (ITA) has been extensively explored in the literature. However, the use of LLMs to support ITA has often been opportunistic, relying on ad hoc prompt engineering (PE) approaches, thereby undermining the reliability, transparency, and replicability of the analysis. The goal of this study is to develop a structured approach to PE in LLM-assisted ITA. To this end, a comprehensive review of the existing literature is conducted to examine how ITA researchers integrate LLMs into their workflows and, in particular, how PE is utilized to support the analytical process. Built on the insights generated from this review, four key steps for effective PE in LLM-assisted ITA are identified and extensively outlined. Furthermore, the study explores state-of-the-art PE techniques that can enhance the execution of these steps, providing ITA researchers with practical strategies to improve their analyses. In conclusion, the main contributions of this paper include: (i) it maps the existing research on LLM-assisted ITA to enable a better understanding of the rapidly developing field, (ii) it outlines a structured four-step PE process to enhance methodological rigor, (iii) it discusses the application of advanced PE techniques to support the execution of these steps, and (iv) it highlights key directions for future research.

Paper Structure

This paper contains 30 sections, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. An Introduction to Prompt Engineering
  3. Systematic Literature Review
  4. Planning
  5. Conducting
  6. Results
  7. DEQ1: What is the context in which the study is conducted and what data are used?
  8. DEQ2: What are the primary objectives and limitations of the study ?
  9. DEQ3: What LLM was used to conduct the analysis?
  10. DEQ4: How does the study evaluate the performance of LLM in aiding ITA?
  11. DEQ5: How does the study define the prompts used to conduct LLM-assisted ITA?
  12. DEQ6: Does the study explain how model responses were refined?
  13. Analysis
  14. The Four-Step PE Process
  15. Step 1: Devise task completion strategy
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Study selection process.
  • Figure 2: Prompt Engineering (PE) Process.