Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collaborative Human-AI Risk Annotation: Co-Annotating Online Incivility with CHAIRA

Jinkyung Katie Park, Rahul Dev Ellezhuthil, Pamela Wisniewski, Vivek Singh

TL;DR

The study reveals the benefits and challenges of using AI agents for incivility annotation and provides design implications and best practices for human-AI collaboration in subjective data annotation.

Abstract

Collaborative human-AI annotation is a promising approach for various tasks with large-scale and complex data. Tools and methods to support effective human-AI collaboration for data annotation are an important direction for research. In this paper, we present CHAIRA: a Collaborative Human-AI Risk Annotation tool that enables human and AI agents to collaboratively annotate online incivility. We leveraged Large Language Models (LLMs) to facilitate the interaction between human and AI annotators and examine four different prompting strategies. The developed CHAIRA system combines multiple prompting approaches with human-AI collaboration for online incivility data annotation. We evaluated CHAIRA on 457 user comments with ground truth labels based on the inter-rater agreement between human and AI coders. We found that the most collaborative prompt supported a high level of agreement between a human agent and AI, comparable to that of two human coders. While the AI missed some implicit incivility that human coders easily identified, it also spotted politically nuanced incivility that human coders overlooked. Our study reveals the benefits and challenges of using AI agents for incivility annotation and provides design implications and best practices for human-AI collaboration in subjective data annotation.

Collaborative Human-AI Risk Annotation: Co-Annotating Online Incivility with CHAIRA

TL;DR

The study reveals the benefits and challenges of using AI agents for incivility annotation and provides design implications and best practices for human-AI collaboration in subjective data annotation.

Abstract

Collaborative human-AI annotation is a promising approach for various tasks with large-scale and complex data. Tools and methods to support effective human-AI collaboration for data annotation are an important direction for research. In this paper, we present CHAIRA: a Collaborative Human-AI Risk Annotation tool that enables human and AI agents to collaboratively annotate online incivility. We leveraged Large Language Models (LLMs) to facilitate the interaction between human and AI annotators and examine four different prompting strategies. The developed CHAIRA system combines multiple prompting approaches with human-AI collaboration for online incivility data annotation. We evaluated CHAIRA on 457 user comments with ground truth labels based on the inter-rater agreement between human and AI coders. We found that the most collaborative prompt supported a high level of agreement between a human agent and AI, comparable to that of two human coders. While the AI missed some implicit incivility that human coders easily identified, it also spotted politically nuanced incivility that human coders overlooked. Our study reveals the benefits and challenges of using AI agents for incivility annotation and provides design implications and best practices for human-AI collaboration in subjective data annotation.
Paper Structure (29 sections, 5 figures, 5 tables)

This paper contains 29 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Conversational Agents as Annotators
  4. Human-AI Collaboration on Annotation
  5. Methods
  6. Design and System Implementation of CHAIRA
  7. System Implementation and Dataset
  8. Design of Web Interface
  9. Creating Prompts
  10. Evaluating Inter-rater Agreement
  11. Human-AI Interaction
  12. Prompt Engineering Approaches
  13. Zero-shot prompting
  14. Definition prompting
  15. Few-shot prompting
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Overview of CHAIRA web interface. A list of designed prompts is shown on the left side, while the prompt, conversation log, and evaluation metrics/results are shown on the right side.
  • Figure 2: Features to create and manage prompts to interact with CHAIRA
  • Figure 3: Features to assess inter-rater reliability between the human coders and CHAIRA
  • Figure 4: Features to support interactive communication between the human coders and CHAIRA
  • Figure 5: Pipeline of Two-stage Chain of Thought prompting. Human feedback on Phase 1 and CA responses are prepended to the input for Phase 2 sent to CA.