The Rise of Parameter Specialization for Knowledge Storage in Large Language Models

TL;DR

This work investigates how knowledge is stored in large language models, focusing on MLP parameters as key-value memories and the phenomenon of knowledge superposition. It introduces SpecWiki, a 525-concept encyclopedic probe, and a masking-based framework to quantify Parameter Specialization Score (PSS) by evaluating concept-specific versus general knowledge contributions. Across 20 open-source models, the study finds that stronger models exhibit higher parameter specialization, with PSS correlating with SpecWiki performance and increasing with model scale and exposure during pretraining. Causal finetuning experiments further show that concentrating related knowledge into focused MLP vectors improves knowledge-task accuracy and reduces hallucination, underscoring the importance of aligning knowledge storage with retrieval pathways for efficiency and reliability.

Abstract

Over time, a growing wave of large language models from various series has been introduced to the community. Researchers are striving to maximize the performance of language models with constrained parameter sizes. However, from a microscopic perspective, there has been limited research on how to better store knowledge in model parameters, particularly within MLPs, to enable more effective utilization of this knowledge by the model. In this work, we analyze twenty publicly available open-source large language models to investigate the relationship between their strong performance and the way knowledge is stored in their corresponding MLP parameters. Our findings reveal that as language models become more advanced and demonstrate stronger knowledge capabilities, their parameters exhibit increased specialization. Specifically, parameters in the MLPs tend to be more focused on encoding similar types of knowledge. We experimentally validate that this specialized distribution of knowledge contributes to improving the efficiency of knowledge utilization in these models. Furthermore, by conducting causal training experiments, we confirm that this specialized knowledge distribution plays a critical role in improving the model's efficiency in leveraging stored knowledge.

Figures (6)

• Figure 1: Evolution of knowledge distribution in model parameters during three iterations of LLaMA models. Each parameter vector corresponds to a column in the value matrix of the MLP module, as indicated by the dashed rectangles.
• Figure 2: Correlation between the performance on SpecWiki and parameter specialization score (PSS) in 20 language models. We use a color gradient to distinguish the release times of the models, with cooler colors indicating earlier release dates and warmer colors representing later releases. Additionally, the size of each circle reflects the model's performance on MMLU, with larger circles indicating better performance. The blue trendline, obtained through linear regression fitting of the data points, suggests a strong correlation between a model’s performance on SpecWiki and its degree of Parameter Specialization.
• Figure 3: Analysis of Parameter Specialization variations across models within the same family. We selected eight models from four model families: LLaMA, Qwen, Mistral, and Gemma. The figure shows how the difference between the General Score (representing the model's ability to handle irrelevant knowledge) and the Concept Specific Score (representing the model's ability to handle task-specific knowledge) changes under different masking ratios of parameter vectors.
• Figure 4: Development of Parameter Specialization in OLMo-2-1124-7B over the pretraining process.
• Figure 5: Relationship between concept popularity, model accuracy on MCQ, and Parameter Specialization Score in LLaMA2-7B and Qwen2-7B models.