Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models

TL;DR

Next Concept Prediction (NCP) shifts language model pretraining from token-level prediction to a discrete, concept-level objective, aiming to leverage higher-level abstractions. ConceptLM combines a token-level encoder with a concept-level module that uses Vector Quantization to form a discrete concept vocabulary of size $N^S$ across $S$ segments, compressing sequences by a factor of $k$ and predicting the next concept to guide token generation. The model jointly optimizes $\,\mathcal{L}_{\text{NTP}}$, $\mathcal{L}_{\text{NCP}}$, and $\mathcal{L}_{\text{VQ}}$, enabling end-to-end training that improves long-range modeling and scaling behavior. Empirical results show ConceptLM outperforms token-level baselines across GPT-2 and Pythia scales and yields additional gains via continual pretraining on 8B-parameter Llama, suggesting a promising path toward stronger, more data-efficient LLMs.

Abstract

We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13 benchmarks show that NCP yields consistent performance gains over traditional token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful language models by introducing a harder pretraining task, providing a promising path toward better language modeling.

Figures (5)

• Figure 1: Scaling performance comparison of ConceptLM against traditional token-level models across three key dimensions: model parameters, training tokens, and total FLOPs. The results demonstrate that ConceptLM exhibits superior scalability.
• Figure 2: Overview of ConceptLM. The figure is organized into three components: the left one shows the ConceptLM workflow; the central part shows the concept prediction process; and the right one shows the segmented VQ strategy used to train the concept vocabulary. We only quantize the concept-level representations to train the codebooks in the training phase.
• Figure 3: Forward and backward of ConceptLM.
• Figure 4: Long-range dependency experiments.
• Figure 5: Codebook usage. The figure shows the usage frequency of each code entry across 12 codebooks, each containing 1,024 entries, evaluated on the OpenWebText test set.