Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique

Zackary Okun Dunivin, Mobina Noori, Seth Frey, Curtis Atkinson

TL;DR

This work tackles the reliability limitations of zero- and few-shot LLM-based qualitative coding by introducing a two-stage workflow: a first-pass annotator applies a human-guided, LLM-adapted codebook, followed by a separate LLM critic that self-reflects on stage-one rationales to veto incorrect positives. The approach is evaluated on six codes across 3,000 ASF project-evaluation emails, with an error taxonomy identifying misinterpretation and meta-discussion as core failure modes. Across three evaluation phases, the secondary critique yields notable gains in deployment-oriented metrics, particularly for codes with high initial false-positive rates, increasing F1 by up to ~0.25 and κ by ~0.26 in targeted cases. The study demonstrates a bounded-autonomy, human-guided self-reflection that can be integrated with existing LLM-assisted annotation pipelines to reduce noise, improve precision, and maintain interpretive alignment with research objectives. The findings highlight the value of post-hoc error analysis and codebook refinement as essential components of scalable, transparent qualitative coding with LLMs.

Abstract

Large language models (LLMs) allow for sophisticated qualitative coding of large datasets, but zero- and few-shot classifiers can produce an intolerable number of errors, even with careful, validated prompting. We present a simple, generalizable two-stage workflow: an LLM applies a human-designed, LLM-adapted codebook; a secondary LLM critic performs self-reflection on each positive label by re-reading the source text alongside the first model's rationale and issuing a final decision. We evaluate this approach on six qualitative codes over 3,000 high-content emails from Apache Software Foundation project evaluation discussions. Our human-derived audit of 360 positive annotations (60 passages by six codes) found that the first-line LLM had a false-positive rate of 8% to 54%, despite F1 scores of 0.74 and 1.00 in testing. Subsequent recoding of all stage-one annotations via a second self-reflection stage improved F1 by 0.04 to 0.25, bringing two especially poor performing codes up to 0.69 and 0.79 from 0.52 and 0.55 respectively. Our manual evaluation identified two recurrent error classes: misinterpretation (violations of code definitions) and meta-discussion (debate about a project evaluation criterion mistaken for its use as a decision justification). Code-specific critic clauses addressing observed failure modes were especially effective with testing and refinement, replicating the codebook-adaption process for LLM interpretation in stage-one. We explain how favoring recall in first-line LLM annotation combined with secondary critique delivers precision-first, compute-light control. With human guidance and validation, self-reflection slots into existing LLM-assisted annotation pipelines to reduce noise and potentially salvage unusable classifiers.

Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique

TL;DR

This work tackles the reliability limitations of zero- and few-shot LLM-based qualitative coding by introducing a two-stage workflow: a first-pass annotator applies a human-guided, LLM-adapted codebook, followed by a separate LLM critic that self-reflects on stage-one rationales to veto incorrect positives. The approach is evaluated on six codes across 3,000 ASF project-evaluation emails, with an error taxonomy identifying misinterpretation and meta-discussion as core failure modes. Across three evaluation phases, the secondary critique yields notable gains in deployment-oriented metrics, particularly for codes with high initial false-positive rates, increasing F1 by up to ~0.25 and κ by ~0.26 in targeted cases. The study demonstrates a bounded-autonomy, human-guided self-reflection that can be integrated with existing LLM-assisted annotation pipelines to reduce noise, improve precision, and maintain interpretive alignment with research objectives. The findings highlight the value of post-hoc error analysis and codebook refinement as essential components of scalable, transparent qualitative coding with LLMs.

Abstract

Large language models (LLMs) allow for sophisticated qualitative coding of large datasets, but zero- and few-shot classifiers can produce an intolerable number of errors, even with careful, validated prompting. We present a simple, generalizable two-stage workflow: an LLM applies a human-designed, LLM-adapted codebook; a secondary LLM critic performs self-reflection on each positive label by re-reading the source text alongside the first model's rationale and issuing a final decision. We evaluate this approach on six qualitative codes over 3,000 high-content emails from Apache Software Foundation project evaluation discussions. Our human-derived audit of 360 positive annotations (60 passages by six codes) found that the first-line LLM had a false-positive rate of 8% to 54%, despite F1 scores of 0.74 and 1.00 in testing. Subsequent recoding of all stage-one annotations via a second self-reflection stage improved F1 by 0.04 to 0.25, bringing two especially poor performing codes up to 0.69 and 0.79 from 0.52 and 0.55 respectively. Our manual evaluation identified two recurrent error classes: misinterpretation (violations of code definitions) and meta-discussion (debate about a project evaluation criterion mistaken for its use as a decision justification). Code-specific critic clauses addressing observed failure modes were especially effective with testing and refinement, replicating the codebook-adaption process for LLM interpretation in stage-one. We explain how favoring recall in first-line LLM annotation combined with secondary critique delivers precision-first, compute-light control. With human guidance and validation, self-reflection slots into existing LLM-assisted annotation pipelines to reduce noise and potentially salvage unusable classifiers.
Paper Structure (19 sections, 2 equations, 3 figures, 5 tables)

This paper contains 19 sections, 2 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Codebook development and first-stage coding
  4. Error analysis: developing a classification schema for false positives
  5. Prompting strategy for secondary LLM critique
  6. Full pipeline evaluation design
  7. Results
  8. Error analysis of stage-one positives and stage-two prompt validation
  9. Full pipeline performance after secondary critique
  10. Discussion
  11. Designing LLM-annotation strategy for compatibility with critique coding
  12. Developing an error taxonomy
  13. The role of iterative and post-hoc error analyses in LLM-assisted qualitative analysis
  14. Conclusion
  15. Code Descriptions
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Two-stage annotation pipeline with three evaluation phases. While the pipeline summarizes model execution on the corpus, the evaluation phases document the human-guided process of prompt development, error analysis, and validation that makes the workflow reliable in practice.
  • Figure 2: First and second layer of the prompt for secondary LLM critique for the Mentor Engagement code.
  • Figure 3: Third layer of the prompt for secondary LLM critique including the chain-of-thought justification, formatting instructions, and original passage and annotation for critique.