Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

Payal Bajaj, Chenyan Xiong, Guolin Ke, Xiaodong Liu, Di He, Saurabh Tiwary, Tie-Yan Liu, Paul Bennett, Xia Song, Jianfeng Gao

TL;DR

METRO presents an efficient denoising pretraining recipe for large-scale autoencoding LMs by using an auxiliary model to generate training signals, extending ELECTRA with curriculum-based denoising and a suite of efficiency/stability techniques. Through systematic empirical exploration, METRO-LM configurations are optimized and scaled from Base to XXL (>5B parameters) using ZeRO, fused CUDA ops, and stability measures, achieving SOTA results on GLUE, SuperGLUE, and SQuAD while reducing pretraining cost. The work demonstrates strong scaling efficiency, showing that smaller, cheaper METRO-LMs can outperform larger predecessors, and provides a stable fine-tuning approach (PDR+MT-DNN) for billion-parameter models. Overall, METRO offers a practical path to high-performing, energy-efficient autoencoding LMs at scale, with detailed guidance on architecture choices, training objectives, and optimization strategies.

Abstract

We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training strategy has demonstrated sample-efficiency to pretrain models at the scale of hundreds of millions of parameters. In this work, we conduct a comprehensive empirical study, and propose a recipe, namely "Model generated dEnoising TRaining Objective" (METRO), which incorporates some of the best modeling techniques developed recently to speed up, stabilize, and enhance pretrained language models without compromising model effectiveness. The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks. More importantly, METRO-LM are efficient in that they often outperform previous large models with significantly smaller model sizes and lower pretraining cost.

METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

TL;DR

METRO presents an efficient denoising pretraining recipe for large-scale autoencoding LMs by using an auxiliary model to generate training signals, extending ELECTRA with curriculum-based denoising and a suite of efficiency/stability techniques. Through systematic empirical exploration, METRO-LM configurations are optimized and scaled from Base to XXL (>5B parameters) using ZeRO, fused CUDA ops, and stability measures, achieving SOTA results on GLUE, SuperGLUE, and SQuAD while reducing pretraining cost. The work demonstrates strong scaling efficiency, showing that smaller, cheaper METRO-LMs can outperform larger predecessors, and provides a stable fine-tuning approach (PDR+MT-DNN) for billion-parameter models. Overall, METRO offers a practical path to high-performing, energy-efficient autoencoding LMs at scale, with detailed guidance on architecture choices, training objectives, and optimization strategies.

Abstract

We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. Originated in ELECTRA, this training strategy has demonstrated sample-efficiency to pretrain models at the scale of hundreds of millions of parameters. In this work, we conduct a comprehensive empirical study, and propose a recipe, namely "Model generated dEnoising TRaining Objective" (METRO), which incorporates some of the best modeling techniques developed recently to speed up, stabilize, and enhance pretrained language models without compromising model effectiveness. The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks. More importantly, METRO-LM are efficient in that they often outperform previous large models with significantly smaller model sizes and lower pretraining cost.
Paper Structure (39 sections, 9 equations, 5 figures, 13 tables, 2 algorithms)

This paper contains 39 sections, 9 equations, 5 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pretraining Models and Techniques
  4. Denoising Pretraining
  5. Rule-based Corruption.
  6. Model-based Corruption.
  7. Construction of METRO-LM
  8. Baseline Model
  9. Baseline Training Objectives.
  10. Baseline Configurations.
  11. Candidate Techniques to Integrate
  12. Transformer Architectures.
  13. Training Objectives.
  14. Dropout.
  15. Empirical Explorations for Best METRO-LM Configuration
  16. ...and 24 more sections

Figures (5)

  • Figure 1: The growth of autoencoding language models and their performances on MNLI test set.
  • Figure 2: Training autoencoders to denoise inputs corrupted by rules versus by auxiliary models.
  • Figure 3: Training curves of first 60K pretraining steps of METRO-LM$_\text{XL}$ with different optimization configurations. Replace Rate is the fraction of tokens replaced by the auxiliary model (out of all tokens). Replace Accuracy is the fraction of replaced tokens detected by the main model.
  • Figure 4: Score distribution on MNLI-m/mm dev average of METRO-LM$_\text{XXL}$ with and without Posterior Differential Regularization (PDR) and Data Augmentation (DA).
  • Figure 5: Model performances on the dev set of MNLI-m and SQuAD F1 w.r.t. model sizes, marked in logarithmic by x-axes (a, b), and at different pretraining computing cost (total teraFLOPs at different percentages of pretraining steps) on x-axes of (c, d).