Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RexBERT: Context Specialized Bidirectional Encoders for E-commerce

Rahul Bajaj, Anuj Garg

TL;DR

RexBERT tackles the gap between generic encoders and domain-specific e-commerce needs by building Ecom-niverse, a domain-focused corpus, and a three-phase pretraining curriculum that includes long-context extension to $8{,}192$ tokens and Guided MLM. Trained across four sizes (from 17M to 400M parameters), RexBERT achieves state-of-the-art or competitive results on domain tasks such as token classification and semantic similarity, while also performing well on general NLU benchmarks like GLUE. The study demonstrates that high-quality in-domain data and a principled curriculum can yield gains comparable to, or exceeding, larger general-purpose models, with practical benefits for retrieval, ranking, and catalog processing. The approach is modular and transferable, providing a blueprint for building domain-specific encoders in other verticals beyond e-commerce.

Abstract

Encoder-only transformers remain indispensable in retrieval, classification, and ranking systems where latency, stability, and cost are paramount. Most general purpose encoders, however, are trained on generic corpora with limited coverage of specialized domains. We introduce RexBERT, a family of BERT-style encoders designed specifically for e-commerce semantics. We make three contributions. First, we release Ecom-niverse, a 350 billion token corpus curated from diverse retail and shopping sources. We describe a modular pipeline that isolates and extracts e-commerce content from FineFineWeb and other open web resources, and characterize the resulting domain distribution. Second, we present a reproducible pretraining recipe building on ModernBERT's architectural advances. The recipe consists of three phases: general pre-training, context extension, and annealed domain specialization. Third, we train RexBERT models ranging from 17M to 400M parameters and evaluate them on token classification, semantic similarity, and general natural language understanding tasks using e-commerce datasets. Despite having 2-3x fewer parameters, RexBERT outperforms larger general-purpose encoders and matches or surpasses modern long-context models on domain-specific benchmarks. Our results demonstrate that high quality in-domain data combined with a principled training approach provides a stronger foundation for e-commerce applications than indiscriminate scaling alone.

RexBERT: Context Specialized Bidirectional Encoders for E-commerce

TL;DR

RexBERT tackles the gap between generic encoders and domain-specific e-commerce needs by building Ecom-niverse, a domain-focused corpus, and a three-phase pretraining curriculum that includes long-context extension to tokens and Guided MLM. Trained across four sizes (from 17M to 400M parameters), RexBERT achieves state-of-the-art or competitive results on domain tasks such as token classification and semantic similarity, while also performing well on general NLU benchmarks like GLUE. The study demonstrates that high-quality in-domain data and a principled curriculum can yield gains comparable to, or exceeding, larger general-purpose models, with practical benefits for retrieval, ranking, and catalog processing. The approach is modular and transferable, providing a blueprint for building domain-specific encoders in other verticals beyond e-commerce.

Abstract

Encoder-only transformers remain indispensable in retrieval, classification, and ranking systems where latency, stability, and cost are paramount. Most general purpose encoders, however, are trained on generic corpora with limited coverage of specialized domains. We introduce RexBERT, a family of BERT-style encoders designed specifically for e-commerce semantics. We make three contributions. First, we release Ecom-niverse, a 350 billion token corpus curated from diverse retail and shopping sources. We describe a modular pipeline that isolates and extracts e-commerce content from FineFineWeb and other open web resources, and characterize the resulting domain distribution. Second, we present a reproducible pretraining recipe building on ModernBERT's architectural advances. The recipe consists of three phases: general pre-training, context extension, and annealed domain specialization. Third, we train RexBERT models ranging from 17M to 400M parameters and evaluate them on token classification, semantic similarity, and general natural language understanding tasks using e-commerce datasets. Despite having 2-3x fewer parameters, RexBERT outperforms larger general-purpose encoders and matches or surpasses modern long-context models on domain-specific benchmarks. Our results demonstrate that high quality in-domain data combined with a principled training approach provides a stronger foundation for e-commerce applications than indiscriminate scaling alone.
Paper Structure (27 sections, 1 equation, 4 figures, 4 tables)

This paper contains 27 sections, 1 equation, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Encoder‑only transformers
  5. Domain‑specific encoders
  6. Other domain‑specific BERT models
  7. E‑commerce‑specific encoders
  8. Encoder vs. decoder studies
  9. Corpus Curation
  10. Source corpus and cleaning pipeline.
  11. Training Methodology
  12. Tokenizer and optimizer.
  13. Phase 1: general pre‑training
  14. Phase 2: context extension
  15. Phase 3: annealing
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Domain distribution of Ecom‑niverse. Sizes represent the amount of filtered data contributed by each FineFineWeb domain and additional corpora. Hobby and News supply the largest portions.
  • Figure 2: Ecom-niverse curation pipeline: domain selection, sampling, LLM labeling, QA auditing, fastText distillation, and thresholded filtering at scale.
  • Figure 3: Training curriculum for RexBERT. The model first trains on 1.7 trillion tokens of mixed data, then extends the context to 8,192 tokens for an additional 250 billion tokens, and finally anneals onto the Ecom‑niverse corpus for 350 billion tokens.
  • Figure 4: Spearman correlation on the ESCI semantic similarity task. The RexBERT series (Micro to Large) achieves higher correlation than ModernBERT, Ettin and EmbeddingGemma models.