Beyond Tokens: Concept-Level Training Objectives for LLMs

TL;DR

The paper addresses the misalignment between next-token prediction and semantic understanding in LLMs by proposing Next Concept Prediction (NCP), which trains models to predict concept units rather than surface tokens. It introduces concept resolution and extraction pipelines (synonyms and hypernyms via WordNet and context-aware prompts) and implements two post-training paradigms (Data Augmentation and Loss Function) across multiple domains. Empirical results show that NCP yields lower perplexities, improved cross-domain robustness, and superior performance on seven downstream benchmarks compared to NTP baselines, suggesting concept-level supervision as a more human-aligned foundation for LLMs. The work points to broader implications for improving abstraction, reasoning, and transfer in LLMs, with future directions including pre-training integration, hierarchical concepts, and multilingual extensions.

Abstract

The next-token prediction (NTP) objective has been foundational in the development of modern large language models (LLMs), driving advances in fluency and generalization. However, NTP operates at the \textit{token} level, treating deviations from a single reference continuation as errors even when alternative continuations are equally plausible or semantically equivalent (e.g., ``mom'' vs. ``mother''). As a result, token-level loss can penalize valid abstractions, paraphrases, or conceptually correct reasoning paths, biasing models toward surface form rather than underlying meaning. This mismatch between the training signal and semantic correctness motivates learning objectives that operate over higher-level representations. We propose a shift from token-level to concept-level prediction, where concepts group multiple surface forms of the same idea (e.g., ``mom,'' ``mommy,'' ``mother'' $\rightarrow$ \textit{MOTHER}). We introduce various methods for integrating conceptual supervision into LLM training and show that concept-aware models achieve lower perplexity, improved robustness under domain shift, and stronger performance than NTP-based models on diverse NLP benchmarks. This suggests \textit{concept-level supervision} as an improved training signal that better aligns LLMs with human semantic abstractions.

Figures (1)

• Figure 1: An outline of our method. We extract context-dependent and independent synonyms and hypernyms and use them to train our NSP and NHP models. We upsample the original data for the NTP baselines. All models are fine-tuned on benchmarks.