From Quotes to Concepts: Axial Coding of Political Debates with Ensemble LMs

TL;DR

This work addresses scaling axial coding in qualitative analysis by operationalizing it with ensemble LLMs. It introduces two axial-coding strategies—density-based clustering with LLM labeling and direct LLM grouping—and evaluates them on Dutch parliamentary debates using both extrinsic topic-alignment metrics and intrinsic qualitative measures. The authors provide a dual contribution: (i) a first scalable axial-coding framework that combines open coding with two axial-grouping approaches, and (ii) a comprehensive evaluation scheme plus a publicly released 5,000-item test dataset to enable reproducibility. The findings reveal a coverage–alignment trade-off: clustering yields broad, interpretable structure with high coverage, while LLM grouping offers concise, semantically aligned categories with stronger topic-fit but lower coverage; a hybrid pipeline is proposed to harness the strengths of both approaches for scalable, interpretable qualitative analysis in IR settings.

Abstract

Axial coding is a commonly used qualitative analysis method that enhances document understanding by organizing sentence-level open codes into broader categories. In this paper, we operationalize axial coding with large language models (LLMs). Extending an ensemble-based open coding approach with an LLM moderator, we add an axial coding step that groups open codes into higher-order categories, transforming raw debate transcripts into concise, hierarchical representations. We compare two strategies: (i) clustering embeddings of code-utterance pairs using density-based and partitioning algorithms followed by LLM labeling, and (ii) direct LLM-based grouping of codes and utterances into categories. We apply our method to Dutch parliamentary debates, converting lengthy transcripts into compact, hierarchically structured codes and categories. We evaluate our method using extrinsic metrics aligned with human-assigned topic labels (ROUGE-L, cosine, BERTScore), and intrinsic metrics describing code groups (coverage, brevity, coherence, novelty, JSD divergence). Our results reveal a trade-off: density-based clustering achieves high coverage and strong cluster alignment, while direct LLM grouping results in higher fine-grained alignment, but lower coverage 20%. Overall, clustering maximizes coverage and structural separation, whereas LLM grouping produces more concise, interpretable, and semantically aligned categories. To support future research, we publicly release the full dataset of utterances and codes, enabling reproducibility and comparative studies.

Figures (3)

• Figure 1: Pipeline for open and axial coding. In the open coding stage (orange box), we produce open codes for utterances from transcripts (20k train set for finding best parameters, 5k held-out set), using an ensemble of LoRA-finetuned LLMs with a moderator model, following the framework of parfenova-pfeffer-2025-measuring. The axial coding stage (pink box) groups them into categories using one of two methods: direct grouping with LLM prompting (path A), or clustering embeddings, followed by LLM labeling (path B). Evaluation (blue box) covers both extrinsic alignment with human-assigned domain labels and intrinsic interpretability metrics (coverage, brevity, coherence, novelty, divergence). The output (green box) is a structured mapping from utterances to codes and categories, see also Fig. \ref{['fig:axial_coding_hierarchy']}.
• Figure 2: Example concept map based on the subset of results with HDBSCAN on the 5k held-out set. Utterances (grey) → open codes (green) → first-order axial categories (yellow) → second-order aggregates (blue). The second-order aggregates were obtained by applying the same grouping on the first-order categories; this is illustrative (not used in the evaluation), but helps convey the hierarchical nature of the axial coding process.
• Figure 3: Coverage vs. cosine similarity for clustering- and LLM-based axial coding methods. Pink triangles denote LLM-based grouping, green crosses denote DBSCAN variants, and orange crosses HDBSCAN variants.