Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models

Patrick Haller, Jonas Golde, Alan Akbik

TL;DR

The paper investigates sample-efficient language modeling in low-resource settings by revisiting RNN-based architectures, notably HGRN2, as alternatives to transformers. It introduces BabyHGRN, an HGRN2-based LM with approximately 330M parameters, trained on BabyLM-scale data and a Pile-derived corpus, and enhanced with knowledge distillation. Through extensive experiments, BabyHGRN is shown to be competitive with, and often superior to, transformer baselines and other subquadratic RNNs on benchmarks such as BLiMP, EWoK, GLUE/SuperGLUE, and BEAR, especially under data-constrained conditions. The results highlight the viability of subquadratic RNN architectures for sample-efficient language modeling and demonstrate the added benefit of knowledge distillation in low-resource regimes.

Abstract

This paper explores the potential of recurrent neural networks (RNNs) and other subquadratic architectures as competitive alternatives to transformer-based models in low-resource language modeling scenarios. We utilize HGRN2 (Qin et al., 2024), a recently proposed RNN-based architecture, and comparatively evaluate its effectiveness against transformer-based baselines and other subquadratic architectures (LSTM, xLSTM, Mamba). Our experimental results show that BABYHGRN, our HGRN2 language model, outperforms transformer-based models in both the 10M and 100M word tracks of the challenge, as measured by their performance on the BLiMP, EWoK, GLUE and BEAR benchmarks. Further, we show the positive impact of knowledge distillation. Our findings challenge the prevailing focus on transformer architectures and indicate the viability of RNN-based models, particularly in resource-constrained environments.

BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models

TL;DR

The paper investigates sample-efficient language modeling in low-resource settings by revisiting RNN-based architectures, notably HGRN2, as alternatives to transformers. It introduces BabyHGRN, an HGRN2-based LM with approximately 330M parameters, trained on BabyLM-scale data and a Pile-derived corpus, and enhanced with knowledge distillation. Through extensive experiments, BabyHGRN is shown to be competitive with, and often superior to, transformer baselines and other subquadratic RNNs on benchmarks such as BLiMP, EWoK, GLUE/SuperGLUE, and BEAR, especially under data-constrained conditions. The results highlight the viability of subquadratic RNN architectures for sample-efficient language modeling and demonstrate the added benefit of knowledge distillation in low-resource regimes.

Abstract

This paper explores the potential of recurrent neural networks (RNNs) and other subquadratic architectures as competitive alternatives to transformer-based models in low-resource language modeling scenarios. We utilize HGRN2 (Qin et al., 2024), a recently proposed RNN-based architecture, and comparatively evaluate its effectiveness against transformer-based baselines and other subquadratic architectures (LSTM, xLSTM, Mamba). Our experimental results show that BABYHGRN, our HGRN2 language model, outperforms transformer-based models in both the 10M and 100M word tracks of the challenge, as measured by their performance on the BLiMP, EWoK, GLUE and BEAR benchmarks. Further, we show the positive impact of knowledge distillation. Our findings challenge the prevailing focus on transformer architectures and indicate the viability of RNN-based models, particularly in resource-constrained environments.

Paper Structure

This paper contains 19 sections, 3 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. BabyHGRN
  3. Training Dataset
  4. Training Objectives
  5. Training Details
  6. Empirical Evaluation
  7. Evaluation Datasets
  8. Experiment 1: RNN Architecture Selection
  9. Experiment 2: Learning Dynamics of HGRN2
  10. Experiment 3: Impact of Training Dataset
  11. Experiment 4: BabyHGRN With Knowledge Distillation
  12. Results
  13. Related Work
  14. Related Work
  15. Conclusion
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Performance evaluation of epochs of pretraining, with the macro average at epoch 3 being the highest.
  • Figure 2: Default hyperparameters for fine-tuning on the (Super)Glue tasks.
  • Figure 3: Evaluation results of learning rate sweep over different architectures. Scores are reported as the macro average over the three zero-shot benchmarks BLiMP, BLiMP-Supplement and EWoK.