Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interactive Machine Teaching by Labeling Rules and Instances

Giannis Karamanolakis, Daniel Hsu, Luis Gravano

TL;DR

This paper investigates how to exploit an expert’s limited time to create effective supervision and proposes an interactive learning framework, INTERVAL, that achieves efficiency by automatically extracting candidate rules based on rich patterns, and effectiveness by soliciting expert feedback on both candidate rules and individual instances.

Abstract

Weakly supervised learning aims to reduce the cost of labeling data by using expert-designed labeling rules. However, existing methods require experts to design effective rules in a single shot, which is difficult in the absence of proper guidance and tooling. Therefore, it is still an open question whether experts should spend their limited time writing rules or instead providing instance labels via active learning. In this paper, we investigate how to exploit an expert's limited time to create effective supervision. First, to develop practical guidelines for rule creation, we conduct an exploratory analysis of diverse collections of existing expert-designed rules and find that rule precision is more important than coverage across datasets. Second, we compare rule creation to individual instance labeling via active learning and demonstrate the importance of both across 6 datasets. Third, we propose an interactive learning framework, INTERVAL, that achieves efficiency by automatically extracting candidate rules based on rich patterns (e.g., by prompting a language model), and effectiveness by soliciting expert feedback on both candidate rules and individual instances. Across 6 datasets, INTERVAL outperforms state-of-the-art weakly supervised approaches by 7% in F1. Furthermore, it requires as few as 10 queries for expert feedback to reach F1 values that existing active learning methods cannot match even with 100 queries.

Interactive Machine Teaching by Labeling Rules and Instances

TL;DR

This paper investigates how to exploit an expert’s limited time to create effective supervision and proposes an interactive learning framework, INTERVAL, that achieves efficiency by automatically extracting candidate rules based on rich patterns, and effectiveness by soliciting expert feedback on both candidate rules and individual instances.

Abstract

Weakly supervised learning aims to reduce the cost of labeling data by using expert-designed labeling rules. However, existing methods require experts to design effective rules in a single shot, which is difficult in the absence of proper guidance and tooling. Therefore, it is still an open question whether experts should spend their limited time writing rules or instead providing instance labels via active learning. In this paper, we investigate how to exploit an expert's limited time to create effective supervision. First, to develop practical guidelines for rule creation, we conduct an exploratory analysis of diverse collections of existing expert-designed rules and find that rule precision is more important than coverage across datasets. Second, we compare rule creation to individual instance labeling via active learning and demonstrate the importance of both across 6 datasets. Third, we propose an interactive learning framework, INTERVAL, that achieves efficiency by automatically extracting candidate rules based on rich patterns (e.g., by prompting a language model), and effectiveness by soliciting expert feedback on both candidate rules and individual instances. Across 6 datasets, INTERVAL outperforms state-of-the-art weakly supervised approaches by 7% in F1. Furthermore, it requires as few as 10 queries for expert feedback to reach F1 values that existing active learning methods cannot match even with 100 queries.
Paper Structure (39 sections, 1 equation, 5 figures, 11 tables, 1 algorithm)

This paper contains 39 sections, 1 equation, 5 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Characterization of prevalent patterns in offline machine teaching.
  3. Automatic rule extraction via prompting.
  4. Interactive machine teaching.
  5. Problem Definition and Related Work
  6. Problem Definition
  7. Prior Work
  8. Non-interactive approaches.
  9. Interactive learning with instance feedback.
  10. Interactive learning with rule feedback.
  11. Interactive Machine Teaching with Instance and Rule Feedback
  12. Teacher-Student Co-Training
  13. Querying for Instance Feedback
  14. Instance clustering.
  15. Instance selection.
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Our INTERVAL framework supports interaction on both instances and automatically-extracted rules (e.g., by prompting a large language model) for weakly supervised learning.
  • Figure 2: Precision-coverage scatterplots reporting the precision (x-axis) and coverage (y-axis) of the Teacher. Each data point corresponds to a different Teacher-Student pair and its color indicates the F1 score of the Student.
  • Figure 3: Precision-coverage scatterplots for automatically extracted $n$-grams (grey) and prompt-based rules (red). Grid numbers show the count of $n$-gram/prompt-based rules on the corresponding grid. Prompt-based rules can achieve relatively higher precision and coverage than $n$-gram rules.
  • Figure 4: Performance of interactive methods on Yelp (top) and AGNews (bottom) as a function of budget ($T$). INTERVAL outperforms Active Learning with strongest improvements in low-budget settings (left).
  • Figure : Interactive Machine Teaching