Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TACOMORE: Leveraging the Potential of LLMs in Corpus-based Discourse Analysis with Prompt Engineering

Bingru Li, Han Wang

TL;DR

This paper introduces TACOMORE, a four-principle prompting framework (Task, Context, Model, Reproducibility) with five core prompt elements to address concerns about LLM reliability in corpus-based discourse analysis. It applies TACOMORE to three open COVID-19 abstracts tasks—keywords, collocates, and concordances—using GPT-4o and Gemini variants, and evaluates outputs with new metrics: Accuracy, Ethicality, Reasoning, and Reproducibility. Across experiments and an ablation study, TACOMORE improves prompt quality, reduces output variance, and enhances reproducibility while clarifying SOPs for such qualitative analyses. The framework offers a structured approach for deploying LLMs in automated discourse studies and provides a basis for future enhancements like retrieval-augmented generation and fine-tuning in linguistic research contexts.

Abstract

The capacity of LLMs to carry out automated qualitative analysis has been questioned by corpus linguists, and it has been argued that corpus-based discourse analysis incorporating LLMs is hindered by issues of unsatisfying performance, hallucination, and irreproducibility. Our proposed method, TACOMORE, aims to address these concerns by serving as an effective prompting framework in this domain. The framework consists of four principles, i.e., Task, Context, Model and Reproducibility, and specifies five fundamental elements of a good prompt, i.e., Role Description, Task Definition, Task Procedures, Contextual Information and Output Format. We conduct experiments on three LLMs, i.e., GPT-4o, Gemini-1.5-Pro and Gemini-1.5.Flash, and find that TACOMORE helps improve LLM performance in three representative discourse analysis tasks, i.e., the analysis of keywords, collocates and concordances, based on an open corpus of COVID-19 research articles. Our findings show the efficacy of the proposed prompting framework TACOMORE in corpus-based discourse analysis in terms of Accuracy, Ethicality, Reasoning, and Reproducibility, and provide novel insights into the application and evaluation of LLMs in automated qualitative studies.

TACOMORE: Leveraging the Potential of LLMs in Corpus-based Discourse Analysis with Prompt Engineering

TL;DR

This paper introduces TACOMORE, a four-principle prompting framework (Task, Context, Model, Reproducibility) with five core prompt elements to address concerns about LLM reliability in corpus-based discourse analysis. It applies TACOMORE to three open COVID-19 abstracts tasks—keywords, collocates, and concordances—using GPT-4o and Gemini variants, and evaluates outputs with new metrics: Accuracy, Ethicality, Reasoning, and Reproducibility. Across experiments and an ablation study, TACOMORE improves prompt quality, reduces output variance, and enhances reproducibility while clarifying SOPs for such qualitative analyses. The framework offers a structured approach for deploying LLMs in automated discourse studies and provides a basis for future enhancements like retrieval-augmented generation and fine-tuning in linguistic research contexts.

Abstract

The capacity of LLMs to carry out automated qualitative analysis has been questioned by corpus linguists, and it has been argued that corpus-based discourse analysis incorporating LLMs is hindered by issues of unsatisfying performance, hallucination, and irreproducibility. Our proposed method, TACOMORE, aims to address these concerns by serving as an effective prompting framework in this domain. The framework consists of four principles, i.e., Task, Context, Model and Reproducibility, and specifies five fundamental elements of a good prompt, i.e., Role Description, Task Definition, Task Procedures, Contextual Information and Output Format. We conduct experiments on three LLMs, i.e., GPT-4o, Gemini-1.5-Pro and Gemini-1.5.Flash, and find that TACOMORE helps improve LLM performance in three representative discourse analysis tasks, i.e., the analysis of keywords, collocates and concordances, based on an open corpus of COVID-19 research articles. Our findings show the efficacy of the proposed prompting framework TACOMORE in corpus-based discourse analysis in terms of Accuracy, Ethicality, Reasoning, and Reproducibility, and provide novel insights into the application and evaluation of LLMs in automated qualitative studies.

Paper Structure

This paper contains 18 sections, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. AI Application in Corpus Linguistics
  4. Prompt Engineering
  5. Methodology
  6. Review of SOP in Corpus-based Discourse Analysis Tasks
  7. Corpus Construction
  8. The Prompting Framework
  9. Evaluation Metrics
  10. Experiments
  11. Experiment Setup
  12. Performance Evaluation
  13. Ablation Study
  14. Discussion
  15. Conclusion
  16. ...and 3 more sections

Figures (8)

  • Figure 1: An illustration of comparing LLM performance of an untuned prompt with a prompt with our proposed framework TACOMORE in corpus-based discourse analysis tasks.
  • Figure 2: TACOMORE, prompting framework for corpus-based discourse analysis
  • Figure 3: Evaluation Metrics
  • Figure 4: Excerpt of the prompt for keyword analysis.
  • Figure 5: An example of theme categorization by GPT-4o.
  • ...and 3 more figures