Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

Zinat Ara, Hossein Salemi, Sungsoo Ray Hong, Yasas Senarath, Steve Peterson, Amanda Lee Hughes, Hemant Purohit

TL;DR

A Context interface is designed, which generates aids that help beginners identify potential mistakes and provide the hidden context of the presented tweet, and which has implications for designing future interfaces aiming at closing the knowledge gap among annotators.

Abstract

Data annotation interfaces predominantly leverage ground truth labels to guide annotators toward accurate responses. With the growing adoption of Artificial Intelligence (AI) in domain-specific professional tasks, it has become increasingly important to help beginning annotators identify how their early-stage knowledge can lead to inaccurate answers, which in turn, helps to ensure quality annotations at scale. To investigate this issue, we conducted a formative study involving eight individuals from the field of disaster management, each possessing varying levels of expertise. The goal was to understand the prevalent factors contributing to disagreements among annotators when classifying Twitter messages related to disasters and to analyze their respective responses. Our analysis identified two primary causes of disagreement between expert and beginner annotators: 1) a lack of contextual knowledge or uncertainty about the situation, and 2) the absence of visual or supplementary cues. Based on these findings, we designed a Context interface, which generates aids that help beginners identify potential mistakes and provide the hidden context of the presented tweet. The summative study compares Context design with two widely used designs in data annotation UI, Highlight and Reasoning-based interfaces. We found significant differences between these designs in terms of attitudinal and behavioral data. We conclude with implications for designing future interfaces aiming at closing the knowledge gap among annotators.

Closing the Knowledge Gap in Designing Data Annotation Interfaces for AI-powered Disaster Management Analytic Systems

TL;DR

A Context interface is designed, which generates aids that help beginners identify potential mistakes and provide the hidden context of the presented tweet, and which has implications for designing future interfaces aiming at closing the knowledge gap among annotators.

Abstract

Data annotation interfaces predominantly leverage ground truth labels to guide annotators toward accurate responses. With the growing adoption of Artificial Intelligence (AI) in domain-specific professional tasks, it has become increasingly important to help beginning annotators identify how their early-stage knowledge can lead to inaccurate answers, which in turn, helps to ensure quality annotations at scale. To investigate this issue, we conducted a formative study involving eight individuals from the field of disaster management, each possessing varying levels of expertise. The goal was to understand the prevalent factors contributing to disagreements among annotators when classifying Twitter messages related to disasters and to analyze their respective responses. Our analysis identified two primary causes of disagreement between expert and beginner annotators: 1) a lack of contextual knowledge or uncertainty about the situation, and 2) the absence of visual or supplementary cues. Based on these findings, we designed a Context interface, which generates aids that help beginners identify potential mistakes and provide the hidden context of the presented tweet. The summative study compares Context design with two widely used designs in data annotation UI, Highlight and Reasoning-based interfaces. We found significant differences between these designs in terms of attitudinal and behavioral data. We conclude with implications for designing future interfaces aiming at closing the knowledge gap among annotators.
Paper Structure (42 sections, 4 figures)

This paper contains 42 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Formative study (S1)
  4. Recruitment
  5. Method
  6. Understanding Problem Scope
  7. Ground Truth Collection
  8. Collecting samples from social media:
  9. Performing annotation task:
  10. Disagreement Analysis:
  11. Design Considerations
  12. Revealing hidden message context
  13. Providing insights from past annotation decisions
  14. Emphasizing on class-relevance hints
  15. Providing clues for potential errors
  16. ...and 27 more sections

Figures (4)

  • Figure 1: Providing assistance and inquiring if the tweet belongs to TM class in (a) Highlight interface, (b) Reasoning interface, (c) Context interface
  • Figure 2: Summative study result under S1 situation - (a) Behavioral Accuracy: box plots for accuracy scores of all users for three interfaces, (b) Behavioral Efficiency: box plots for total completion time (in minutes) of all users for three interfaces, (c) Attitudinal Accuracy: box plots for users perceptual accuracy ratings on a scale of 1 to 7, (d) Attitudinal Efficiency: box plots for users perceptual efficiency ratings on a scale of 1 to 7, (e) Attitudinal Knowledge Gap Perception: box plots for users knowledge gap perception ratings on a scale of 1 to 7
  • Figure 3: Prompt for generating reasoning based on the user's feedback
  • Figure 4: Prompts for generating reasoning behind the users' disagreement