Chinese ModernBERT with Whole-Word Masking

TL;DR

Chinese ModernBERT tackles tokenization and long-context efficiency in encoder-only models for Chinese by introducing a hardware-aware $32{,}000$ BPE vocabulary, whole-word masking with a dynamic curriculum, and a two-stage pre-training pipeline that expands native context from $1{,}024$ to $8{,}192$ tokens using RoPE and local/global attention, together with a damped-cosine learning-rate schedule. It achieves competitive CLUE results, high long-context throughput under bf16, and meaningful open-data STS gains on SimCLUE when fine-tuned with modest open data (e.g., SimCLUE ~3M pairs, plus T2Ranking ~2M). The work demonstrates a scalable, reproducible path for Chinese STS and downstream tasks, releasing tokenizer and weights to support research and deployment. Overall, the design choices—hardware-aware vocabulary, dynamic WWM, RoPE-based long-context, and staged pre-training—yield strong efficiency and performance benefits for Chinese NLP in practical settings.

Abstract

Encoder-only Transformers have advanced along three axes -- architecture, data, and systems -- yielding Pareto gains in accuracy, speed, and memory efficiency. Yet these improvements have not fully transferred to Chinese, where tokenization and morphology differ markedly from English. We introduce Chinese ModernBERT, a from-scratch Chinese encoder that couples: (i) a hardware-aware 32k BPE vocabulary tailored to frequent Chinese affixes/compounds, lowering the embedding budget; (ii) whole-word masking (WWM) with a dynamic masking curriculum (30% -> 15%) to align task difficulty with training progress; (iii) a two-stage pre-training pipeline that extends the native context from 1,024 to 8,192 tokens using RoPE and alternating local/global attention; and (iv) a damped-cosine learning-rate schedule for stable long-horizon optimization. We pre-train on ~1.2T Chinese tokens from CCI3-HQ, CCI4 (Chinese), and Cosmopedia-Chinese. On CLUE, Chinese ModernBERT is competitive with strong Chinese encoders under a unified fine-tuning protocol. Under bf16 it achieves high long-sequence throughput while maintaining strong short-sequence speed, reflecting benefits from budget allocation and attention design. To probe retrieval-oriented quality, we add a small amount of open contrastive data: fine-tuning on SimCLUE (~3M pairs) improves further when adding T2Ranking (~2M), reaching 0.505 (Pearson) / 0.537 (Spearman) on the SimCLUE test set. Under this open-data setting, Chinese ModernBERT surpasses Qwen-0.6B-embedding on SimCLUE, suggesting a clear scaling path for STS with additional curated pairs. We will release tokenizer and weights to facilitate reproducible research.