Scaling Embeddings Outperforms Scaling Experts in Language Models

TL;DR

This work investigates embedding scaling as a orthogonal axis to Mixture-of-Experts (MoE) scaling for large language models, identifying regimes where N-gram Embedding improves efficiency relative to increasing experts. It introduces the N-gram Embedding Layer, derives its practical deployment constraints (budgeting, hash collisions, hyperparameters, and amplification), and analyzes how model width and depth influence its effectiveness. The authors validate these insights with LongCat-Flash-Lite, a 68.5B-parameter model with extensive embedding parameters (~$>30$B) that achieves competitive or superior results to parameter-equivalent MoE baselines, particularly in agentic and coding tasks. They further propose system-level optimizations (N-gram Cache, synchronized kernels) and speculative decoding strategies to translate sparsity into real latency and throughput gains, demonstrating robust inference performance and practical deployment potential. Overall, embedding scaling emerges as a promising, scalable augmentation to MoE-based LLMs with tangible speedups and strong downstream capabilities.

Abstract

While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.

Figures (10)

• Figure 1: The architecture of a N-gram Embedding layer huang2025overtokenizedtransformervocabularygenerally. The embedding of each token is augmented by the N-gram Embedding branch.
• Figure 2: The scaling curve of MoE model and N-gram Embedding (NE) model. The horizontal axis is the ratio of total parameters to the activated parameters (280M). The axes of Figures on the right panel is converted to a logarithmic scale. For the two NE curves, we prepend a dashed line to connect the corresponding base MoE model without NE.
• Figure 3: (a) The vocabulary hit rate of different n-grams. (b) The collision number of 2-gram hashing at different vocabulary size. Sampling points are denser near integer multiples of vocabulary size and sparser elsewhere for clarity.
• Figure 4: Comparison of training and validation loss under different combinations of $N$ and $K$.
• Figure 5: Layer-wise analysis of L2 norms for module outputs versus their corresponding identity branches, alongside the ratio of these norms. Each shortcut layer comprises two sub-layers, denoted by suffixes 0 and 1.