Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety

TL;DR

The paper addresses the challenge of enriching NLP datasets with dynamic real-world indicators by introducing the Confidence-Aware Fine-Grained Debate (CFD) framework, where multiple open-source LLMs simulate annotators and exchange fine-grained evidence to reach consensus. CFD uses Cat-CoT for initial per-category decisions, followed by structured, fine-grained debates and two confidence-estimation methods (self-verbalized and sampling-based) to drive consensus or judge decisions, enabling robust data enrichment. The authors validate CFD on two expert-annotated datasets—Life Events for Mental Health and Sharenting for Online Safety—demonstrating that CFD-enriched features improve downstream tasks, with notable gains such as a 10.1% relative improvement in sharenting risk classification. They release the annotated datasets and provide thorough ablation analyses showing that fine-grained confidence and multi-LLM collaboration yield the strongest gains, supporting CFD as a scalable, interpretable approach for domain-specific NLP data enrichment.

Abstract

Real-world indicators are important for improving natural language processing (NLP) tasks such as life events for mental health analysis and risky behaviour for online safety, yet labelling such information in NLP training datasets is often costly and/or difficult given the dynamic nature of such events. This paper compares several LLM-based data enrichment methods and introduces a novel Confidence-Aware Fine-Grained Debate (CFD) framework in which multiple LLM agents simulate human annotators and exchange fine-grained evidence to reach consensus. We describe two new expert-annotated datasets, a mental health Reddit wellbeing dataset and an online safety Facebook sharenting risk dataset. Our CFD framework achieves the most robust data enrichment performance compared to a range of baselines and we show that this type of data enrichment consistently improves downstream tasks. Enriched features incorporated via debate transcripts yield the largest gains, outperforming the non-enriched baseline by 10.1% for the online safety task.

Figures (6)

• Figure 1: Illustration of Confidence-Aware Fine-Grained Debate Framework.
• Figure 2: An example of Categorical CoT output format on life events.
• Figure 3: Comparison of confidence–accuracy monotonicity curves across confidence types and tasks. The top row shows results on the life events task, and the bottom row shows results on the sharenting behaviors task.
• Figure 4: Complementarity across labels in two settings on the life events task.
• Figure 5: Confidence separability gaps ($\Delta_1$ and $\Delta_2$) on the life-events task for single-LLM and multi-LLM multi-agent settings. $\Delta_1$ compares confidence margins between correct$\rightarrow$correct and correct$\rightarrow$wrong cases, while $\Delta_2$ compares wrong$\rightarrow$correct and wrong$\rightarrow$wrong cases. Higher values indicate that confidence provides more discriminative signals between agents when their predictions differ.