A Hybrid Intelligence Method for Argument Mining

TL;DR

HyEnA tackles the problem of extracting diverse, high-quality arguments from large, noisy citizen feedback under tight decision timelines by a novel hybrid human-AI workflow. It splits argument extraction into three phases—annotation, consolidation, and selection—guided by intelligent sampling, pairwise similarity, and clustering to produce cohesive key-arguments; Phase 3 incorporates multiple extraction strategies and LLM prompting to select representative opinions. Across three real-world COVID-19 policy corpora, HyEnA achieves higher precision and diversity than automated baselines and requires fewer opinions than manual expert analyses, while still capturing novel insights. The approach demonstrates the practicality of hybrid intelligence for scalable, accountable policy-relevant summarization of public opinion, with potential for broader application and further integration of advanced LLM-assisted components.

Abstract

Large-scale survey tools enable the collection of citizen feedback in opinion corpora. Extracting the key arguments from a large and noisy set of opinions helps in understanding the opinions quickly and accurately. Fully automated methods can extract arguments but (1) require large labeled datasets that induce large annotation costs and (2) work well for known viewpoints, but not for novel points of view. We propose HyEnA, a hybrid (human + AI) method for extracting arguments from opinionated texts, combining the speed of automated processing with the understanding and reasoning capabilities of humans. We evaluate HyEnA on three citizen feedback corpora. We find that, on the one hand, HyEnA achieves higher coverage and precision than a state-of-the-art automated method when compared to a common set of diverse opinions, justifying the need for human insight. On the other hand, HyEnA requires less human effort and does not compromise quality compared to (fully manual) expert analysis, demonstrating the benefit of combining human and artificial intelligence.

Figures (10)

• Figure 1: In a democratic cycle, citizens provide their opinions on options for governmental decision-making and their opinions need to be interpreted. Insights into the arguments embedded in their comments can be provided by Key Point Analysis (KPA). To perform KPA, most analysis is performed either manually or automatically. In our work, we propose HyEnA, a hybrid method.
• Figure 2: Overview of the HyEnA method.
• Figure 3: Pairwise annotation of the dependency graph, combining human and automatic judgments. Vertices indicate argument pairs; the edge direction points to the argument pair with greater similarity. The highlighted blue edges are a disjoint path selected by the Power algorithm. Iteratively, vertices are annotated as similar (yellow) or non-similar (red).
• Figure 4: Disagreement analysis for the Key Argument Evaluation phase. On the left, argument lengths are the same whether annotators agree or disagree. However, on the right, annotators disagree on match labels in long opinions.
• Figure 5: Distribution of argument overlap ratio for arguments generated by Key Argument Annotation in Phase 1.