Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Data-based Curricula Work?

Maxim K. Surkov, Vladislav D. Mosin, Ivan P. Yamshchikov

TL;DR

This study critically examines data-driven curriculum learning for transformer-based NLP models (e.g., BERT, T5), testing whether ordering or sampling training data by computed complexity can speed training or improve accuracy. It defines several NLP-adapted complexity measures (EE, TSE, length, TF-IDF, max word rank, likelihood) and sampling strategies (CB, HYP, DB, SS, SM), and evaluates them on MLM, text classification, and MT. Across tasks and models, data-driven curricula consistently fail to outperform random or baseline sampling, with many configurations slowing convergence or reducing final quality. The authors argue that without a solid theoretical basis, data-based curricula remain a risky, largely ineffective technique for large-scale transformer training and call for deeper information-theoretic analysis to identify when curriculum ideas can help.

Abstract

Current state-of-the-art NLP systems use large neural networks that require lots of computational resources for training. Inspired by human knowledge acquisition, researchers have proposed curriculum learning, - sequencing of tasks (task-based curricula) or ordering and sampling of the datasets (data-based curricula) that facilitate training. This work investigates the benefits of data-based curriculum learning for large modern language models such as BERT and T5. We experiment with various curricula based on a range of complexity measures and different sampling strategies. Extensive experiments on different NLP tasks show that curricula based on various complexity measures rarely has any benefits while random sampling performs either as well or better than curricula.

Do Data-based Curricula Work?

TL;DR

This study critically examines data-driven curriculum learning for transformer-based NLP models (e.g., BERT, T5), testing whether ordering or sampling training data by computed complexity can speed training or improve accuracy. It defines several NLP-adapted complexity measures (EE, TSE, length, TF-IDF, max word rank, likelihood) and sampling strategies (CB, HYP, DB, SS, SM), and evaluates them on MLM, text classification, and MT. Across tasks and models, data-driven curricula consistently fail to outperform random or baseline sampling, with many configurations slowing convergence or reducing final quality. The authors argue that without a solid theoretical basis, data-based curricula remain a risky, largely ineffective technique for large-scale transformer training and call for deeper information-theoretic analysis to identify when curriculum ideas can help.

Abstract

Current state-of-the-art NLP systems use large neural networks that require lots of computational resources for training. Inspired by human knowledge acquisition, researchers have proposed curriculum learning, - sequencing of tasks (task-based curricula) or ordering and sampling of the datasets (data-based curricula) that facilitate training. This work investigates the benefits of data-based curriculum learning for large modern language models such as BERT and T5. We experiment with various curricula based on a range of complexity measures and different sampling strategies. Extensive experiments on different NLP tasks show that curricula based on various complexity measures rarely has any benefits while random sampling performs either as well or better than curricula.
Paper Structure (18 sections, 7 equations, 6 figures, 4 tables)

This paper contains 18 sections, 7 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Metrics
  3. Samplers
  4. Experiments
  5. Text Classification
  6. Masked Language Modelling
  7. Neural Machine Translation
  8. Discussion
  9. Conclusion
  10. Acknowledgments
  11. Heuristic Approaches to Text Complexity
  12. Using Information Theory for Text Complexity
  13. Additional Experiments
  14. Convergence Speed
  15. Classification
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Pre-trained BERT fine-tuned on Sentiment140 and Hyperpartisan News Detection datasets. Accuracy of the classifier as a function of the number of training steps.
  • Figure 2: Loss function dependency on the number of training steps on MLM for BooksCorpus dataset during the first 40k steps of training. Every plot depicts results for six different complexity estimates combined with a specific sampler.
  • Figure 3: Test results with LSTM on Sentiment140 dataset. Accuracy of the classifier as a function of the number of training steps.
  • Figure 4: Test results for NMT on WMT16 with different learning rates with excess entropy as a complexity measure
  • Figure 5: Test results for NMT on WMT16 with different learning rates with TSE as a complexity measure
  • ...and 1 more figures