Cracking Factual Knowledge: A Comprehensive Analysis of Degenerate Knowledge Neurons in Large Language Models

TL;DR

This work defines Degenerate Knowledge Neurons (DKNs) as structurally and functionally interlinked neuron groups that independently express the same factual knowledge in large language models. It introduces Neurological Topology Clustering (NTC), combining persistent homology and clustering to form DKNs across arbitrary sizes and connections. Through 34 experiments across GPT-2 and LLaMA2-7b, the study links DKNs to robustness, evolvability, and complexity, showing that suppressing or enhancing DKNs can respectively degrade or improve model resilience to input perturbations and fact-checking. The findings demonstrate that DKNs are central to how PLMs learn, store, and refine factual knowledge, with implications for efficient fine-tuning and understanding model capacity, while acknowledging limitations in scale and language scope.

Abstract

Large language models (LLMs) store extensive factual knowledge, but the underlying mechanisms remain unclear. Previous research suggests that factual knowledge is stored within multi-layer perceptron weights, and some storage units exhibit degeneracy, referred to as Degenerate Knowledge Neurons (DKNs). Despite the novelty and unique properties of this concept, it has not been rigorously defined or systematically studied. We first consider the connection weight patterns of MLP neurons and define DKNs from both structural and functional aspects. Based on this, we introduce the Neurological Topology Clustering method, which allows the formation of DKNs in any numbers and structures, leading to a more accurate DKN acquisition. Furthermore, inspired by cognitive science, we explore the relationship between DKNs and the robustness, evolvability, and complexity of LLMs. Our execution of 34 experiments under 6 settings demonstrates the connection between DKNs and these three properties. The code will be available soon.

Figures (16)

• Figure 1: Explanation of KNs and DKNs. (a) illustrates the KNs corresponding to a fact, ($\text{b}_1$) and ($\text{b}_2$) are the preliminary definition of DKNs and our complete definition of DKNs, respectively.
• Figure 2: The clustering part of the Neurological Topology Clustering method. The $x$-axis ($R$) represents the increasing distance threshold starting from 0. Circles with radius $R$ are drawn around neurons, and intersecting circles indicate that the KNs are clustered together.
• Figure 3: The relationship between DKNs (with degeneracy property), robustness, evolvability and complexity in PLMs.
• Figure 4: The relationship between $\Delta Prob$ and the number of suppressed BDCs (using NTC). Table \ref{['table-1-dkn-result']} averages the results for suppressing 1 to n-1 BDCs, while this figure shows the changes as the number of suppressed BDCs varies from 1 to n, with the final point representing all BDCs suppressed. Lower $\Delta Prob$ for partial suppression and higher $\Delta Prob$ for full suppression, i.e., a more prominent final turning point, indicate better degeneracy. As the cardinality of $\mathcal{D}$ varies across PLMs and methods, Table \ref{['table-1-dkn-result']} and Figure \ref{['fig-main']} show representative results. Full results are in Appendix \ref{['appendix:NTC']} (Table \ref{['tab:appendix:NTC']}, Figures \ref{['fig-appendix-main']}, \ref{['fig-appendix-DBSCAN']}, \ref{['fig-appendix-Hierarchical']}, \ref{['fig-appendix-K-Means-1']}, \ref{['fig-appendix-K-Means-2']}).
• Figure 5: Changes in the prediction probabilities of PLMs corresponding to the suppression of DKNs and other baselines. $\mathcal{D}$, $\mathcal{N}$, $Rnd$, and $\emptyset$ represent DKNs, KNs, random neurons, and suppressing no neurons, respectively.