Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Seq vs Seq: An Open Suite of Paired Encoders and Decoders

Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, Benjamin Van Durme

Abstract

The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of the community still uses encoder-only models for tasks such as classification or retrieval. Previous work has attempted to compare these architectures, but is forced to make comparisons with models that have different numbers of parameters, training techniques, and datasets. We introduce the SOTA open-data Ettin suite of models: paired encoder-only and decoder-only models ranging from 17 million parameters to 1 billion, trained on up to 2 trillion tokens. Using the same recipe for both encoder-only and decoder-only models produces SOTA recipes in both categories for their respective sizes, beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as decoders. Like previous work, we find that encoder-only models excel at classification and retrieval tasks while decoders excel at generative tasks. However, we show that adapting a decoder model to encoder tasks (and vice versa) through continued training is subpar compared to using only the reverse objective (i.e. a 400M encoder outperforms a 1B decoder on MNLI, and vice versa for generative tasks). We open-source all artifacts of this study including training data, training order segmented by checkpoint, and 200+ checkpoints to allow future work to analyze or extend all aspects of training.

Seq vs Seq: An Open Suite of Paired Encoders and Decoders

Abstract

The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of the community still uses encoder-only models for tasks such as classification or retrieval. Previous work has attempted to compare these architectures, but is forced to make comparisons with models that have different numbers of parameters, training techniques, and datasets. We introduce the SOTA open-data Ettin suite of models: paired encoder-only and decoder-only models ranging from 17 million parameters to 1 billion, trained on up to 2 trillion tokens. Using the same recipe for both encoder-only and decoder-only models produces SOTA recipes in both categories for their respective sizes, beating ModernBERT as an encoder and Llama 3.2 and SmolLM2 as decoders. Like previous work, we find that encoder-only models excel at classification and retrieval tasks while decoders excel at generative tasks. However, we show that adapting a decoder model to encoder tasks (and vice versa) through continued training is subpar compared to using only the reverse objective (i.e. a 400M encoder outperforms a 1B decoder on MNLI, and vice versa for generative tasks). We open-source all artifacts of this study including training data, training order segmented by checkpoint, and 200+ checkpoints to allow future work to analyze or extend all aspects of training.

Paper Structure

This paper contains 36 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Encoder-only Models
  4. Comparisons between Encoders and Decoders
  5. Bidirectional Attention for Decoders
  6. Checkpoint-Level Model Analyses
  7. Experimental Settings
  8. Training Data
  9. Architecture
  10. Training Recipe
  11. Base Pre-training
  12. Context Extension / Mid-Training
  13. Decay Phase
  14. Major training differences from ModernBERT
  15. Cross-Objective Training
  16. ...and 21 more sections

Figures (3)

  • Figure 1: Encoder vs decoder comparison across model size. Generally models in the preferred architecture (e.g. encoders in MNLI) do better than the opposite architecture even with an order of magnitude greater size, e.g. a 400M decoder outperforming the 1B encoder. Notably, in generative tasks, decoders-from-encoders scale poorly with size.
  • Figure 2: Gender pronoun predictions on the Gotcha split of WinoGender rudinger2018gender, a 50/50 stereotypical male/female split. We see that encoder models are more likely to use gender neutral pronouns whereas both are biased towards male pronouns.
  • Figure 3: Full gender pronoun predictions results on the Gotcha split of WinoGender rudinger2018gender, a 50/50 stereotypical split. We see that encoder models are more likely to use gender neutral pronouns whereas both are biased towards male pronouns.