Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Interactive Human-Machine Learning Interface for Collecting and Learning from Complex Annotations

Jonathan Erskine, Matt Clifford, Alexander Hepburn, Raúl Santos-Rodríguez

TL;DR

This work proposes a human-machine learning interface for binary classification tasks which enables human annotators to utilise counterfactual examples to complement standard binary labels as annotations for a dataset.

Abstract

Human-Computer Interaction has been shown to lead to improvements in machine learning systems by boosting model performance, accelerating learning and building user confidence. In this work, we aim to alleviate the expectation that human annotators adapt to the constraints imposed by traditional labels by allowing for extra flexibility in the form that supervision information is collected. For this, we propose a human-machine learning interface for binary classification tasks which enables human annotators to utilise counterfactual examples to complement standard binary labels as annotations for a dataset. Finally we discuss the challenges in future extensions of this work.

An Interactive Human-Machine Learning Interface for Collecting and Learning from Complex Annotations

TL;DR

This work proposes a human-machine learning interface for binary classification tasks which enables human annotators to utilise counterfactual examples to complement standard binary labels as annotations for a dataset.

Abstract

Human-Computer Interaction has been shown to lead to improvements in machine learning systems by boosting model performance, accelerating learning and building user confidence. In this work, we aim to alleviate the expectation that human annotators adapt to the constraints imposed by traditional labels by allowing for extra flexibility in the form that supervision information is collected. For this, we propose a human-machine learning interface for binary classification tasks which enables human annotators to utilise counterfactual examples to complement standard binary labels as annotations for a dataset. Finally we discuss the challenges in future extensions of this work.
Paper Structure (8 sections, 2 equations, 2 figures)

This paper contains 8 sections, 2 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Methodology
  3. Learning Task
  4. Human-Machine Learning Interface
  5. Leveraging Annotations in Learning
  6. Loss Function
  7. Demonstration
  8. Extension to NLP Domain

Figures (2)

  • Figure 1: A typical machine learning pipeline contrasted with our proposed method.
  • Figure 2: Our Graphical User Interface (GUI) which enables learning from human expert annotations. The current annotation is represented by a green arrow, with prior annotations represented in black. The annotation toolbox (bottom left) enables the annotator to define the direction $\mathbf{d}$ for a selected data point which will lead to a counterfactual observation. The inspection console (bottom-centre) consists of the dataset and the model predicted probabilities across the feature space. The evaluation console (top-centre) enables comparisons across different experiments. This example shows the test accuracy for a control dataset with no annotations and a labelled set which features the counterfactuals directions shown in the inspection console. In both cases the dataset is limited to 9 observations.