Annotator in the Loop: A Case Study of In-Depth Rater Engagement to Create a Bridging Benchmark Dataset

TL;DR

The paper addresses the limitations of isolated crowd annotation and proposes an annotator-in-the-loop approach to produce a high-quality Bridging Benchmark Dataset from 11,973 Civil Comments posts. It combines theory-grounded attribute definitions with iterative, collaborative refinement and weekly human oversight, blending qualitative and quantitative methods to improve inter-rater reliability. The study demonstrates higher IRR using the novel method compared to traditional labeling and emphasizes the value of collective sense-making, contextual understanding, and annotator well-being. This work offers practical guidance for building more reliable prosocial discourse benchmarks with broad implications for safe and interpretable NLP systems.

Abstract

With the growing prevalence of large language models, it is increasingly common to annotate datasets for machine learning using pools of crowd raters. However, these raters often work in isolation as individual crowdworkers. In this work, we regard annotation not merely as inexpensive, scalable labor, but rather as a nuanced interpretative effort to discern the meaning of what is being said in a text. We describe a novel, collaborative, and iterative annotator-in-the-loop methodology for annotation, resulting in a 'Bridging Benchmark Dataset' of comments relevant to bridging divides, annotated from 11,973 textual posts in the Civil Comments dataset. The methodology differs from popular anonymous crowd-rating annotation processes due to its use of an in-depth, iterative engagement with seven US-based raters to (1) collaboratively refine the definitions of the to-be-annotated concepts and then (2) iteratively annotate complex social concepts, with check-in meetings and discussions. This approach addresses some shortcomings of current anonymous crowd-based annotation work, and we present empirical evidence of the performance of our annotation process in the form of inter-rater reliability. Our findings indicate that collaborative engagement with annotators can enhance annotation methods, as opposed to relying solely on isolated work conducted remotely. We provide an overview of the input texts, attributes, and annotation process, along with the empirical results and the resulting benchmark dataset, categorized according to the following attributes: Alienation, Compassion, Reasoning, Curiosity, Moral Outrage, and Respect.