Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Four Quadrants of Difficulty: A Simple Categorisation and its Limits

Vanessa Toborek, Sebastian Müller, Christian Bauckhage

TL;DR

This work introduces a four-quadrant taxonomy of difficulty signals for Curriculum Learning in NLP, distinguishing human vs. model sources and task-agnostic vs. task-dependent information. It empirically tests cross-quadrant interactions on SNLI by computing TA-H, TA-M, TD-H, and TD-M proxies and examining correlations, regressions, and dataset-cartography-style analyses. The results show TA-H signals largely decouple from TD signals, while TD-H and TD-M align; task-dependent predictions from TA features are weak ($R^2$ near 0–0.1). The findings challenge the assumption that linguistic difficulty proxies alone capture model learning difficulty and suggest prioritizing lightweight, task-dependent estimators and scheduler dynamics for effective CL.

Abstract

Curriculum Learning (CL) aims to improve the outcome of model training by estimating the difficulty of samples and scheduling them accordingly. In NLP, difficulty is commonly approximated using task-agnostic linguistic heuristics or human intuition, implicitly assuming that these signals correlate with what neural models find difficult to learn. We propose a four-quadrant categorisation of difficulty signals -- human vs. model and task-agnostic vs. task-dependent -- and systematically analyse their interactions on a natural language understanding dataset. We find that task-agnostic features behave largely independently and that only task-dependent features align. These findings challenge common CL intuitions and highlight the need for lightweight, task-dependent difficulty estimators that better reflect model learning behaviour.

Four Quadrants of Difficulty: A Simple Categorisation and its Limits

TL;DR

This work introduces a four-quadrant taxonomy of difficulty signals for Curriculum Learning in NLP, distinguishing human vs. model sources and task-agnostic vs. task-dependent information. It empirically tests cross-quadrant interactions on SNLI by computing TA-H, TA-M, TD-H, and TD-M proxies and examining correlations, regressions, and dataset-cartography-style analyses. The results show TA-H signals largely decouple from TD signals, while TD-H and TD-M align; task-dependent predictions from TA features are weak ( near 0–0.1). The findings challenge the assumption that linguistic difficulty proxies alone capture model learning difficulty and suggest prioritizing lightweight, task-dependent estimators and scheduler dynamics for effective CL.

Abstract

Curriculum Learning (CL) aims to improve the outcome of model training by estimating the difficulty of samples and scheduling them accordingly. In NLP, difficulty is commonly approximated using task-agnostic linguistic heuristics or human intuition, implicitly assuming that these signals correlate with what neural models find difficult to learn. We propose a four-quadrant categorisation of difficulty signals -- human vs. model and task-agnostic vs. task-dependent -- and systematically analyse their interactions on a natural language understanding dataset. We find that task-agnostic features behave largely independently and that only task-dependent features align. These findings challenge common CL intuitions and highlight the need for lightweight, task-dependent difficulty estimators that better reflect model learning behaviour.
Paper Structure (4 sections, 2 figures, 1 table)

This paper contains 4 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. A Categorisation of Difficulty Signals
  3. Empirical Study: Testing the Quadrant Interactions
  4. Lessons for Curriculum Learning

Figures (2)

  • Figure 1: Pearson correlation for the four difficulty quadrants. (Left) Correlation among task-agnostic human and task-agnostic model difficulty signals. (Right) Correlation for all task-agnostic difficulty signals with all task-dependent ones. All TD-M are averaged over all three models and ten random seeds each.
  • Figure 2: Histograms of easy vs. ambiguous samples (based on the respective model confidence and variability) for selected task-agnostic proxies: length, word rarity, complexity, age of acquisition, and BERT perplexity.