Codebook-Injected Dialogue Segmentation for Multi-Utterance Constructs Annotation: LLM-Assisted and Gold-Label-Free Evaluation

TL;DR

This work tackles the boundary placement problem in dialogue-act annotation by introducing codebook-injected (DA-aware) segmentation that decouples boundary decisions from labeling, enabling multi-utterance constructs to be annotated more faithfully. It evaluates DA-aware and generic LLM-based segmenters, along with coherence-based baselines, using gold-label-free evaluation metrics that capture within-segment coherence, between-segment distinctiveness, and cross-rater distributional agreement across two educational-dialogue datasets. Key findings show that DA-aware LLM segmentation improves intra-segment coherence and construct-consistency, while coherence-based methods better capture global shifts; however, no single method dominates due to trade-offs among coherence, boundary distinctiveness, and human–AI agreement. The results underscore segmentation as a crucial design choice tied to downstream objectives, suggesting LLM-based, DA-aware segmentation for construct-aligned spans and coherence-based methods for detecting sharp dialogue-flow changes, with careful consideration of the downstream annotation goals.

Abstract

Dialogue Act (DA) annotation typically treats communicative or pedagogical intent as localized to individual utterances or turns. This leads annotators to agree on the underlying action while disagreeing on segment boundaries, reducing apparent reliability. We propose codebook-injected segmentation, which conditions boundary decisions on downstream annotation criteria, and evaluate LLM-based segmenters against standard and retrieval-augmented baselines. To assess these without gold labels, we introduce evaluation metrics for span consistency, distinctiveness, and human-AI distributional agreement. We found DA-awareness produces segments that are internally more consistent than text-only baselines. While LLMs excel at creating construct-consistent spans, coherence-based baselines remain superior at detecting global shifts in dialogue flow. Across two datasets, no single segmenter dominates. Improvements in within-segment coherence frequently trade off against boundary distinctiveness and human-AI distributional agreement. These results highlight segmentation as a consequential design choice that should be optimized for downstream objectives rather than a single performance score.