Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Digital Metabolism: Decoupling Logic from Facts via Regenerative Unlearning -- Towards a Pure Neural Logic Core

Mengmeng Peng, Zhenyu Fang, He Sun

TL;DR

This work tackles parameter entanglement in LLMs, where logic and factual knowledge share weights, causing a memory wall and hallucinations. It introduces digital metabolism and the Regenerative Logic-Core Protocol (RLCP), a dual-stream training framework that uses deep-layer gradient reversal to render targeted factual dependencies linearly undecodable while preserving context-driven reasoning. In experiments on Qwen2.5-0.5B-Instruct, RLCP drives factual retention to $<7\%$ accuracy and produces a phase transition toward a structural crystallization of representations, accompanied by emergent chain-of-thought scaffolding on GSM8K. These results suggest a viable, training-based path to decouple facts and logic, with implications for modular neural CPU plus symbolic RAM architectures and improved robustness to hallucination.

Abstract

Large language models (LLMs) currently suffer from parameter entanglement, where general reasoning capabilities (logic) and specific factual knowledge (facts) exist in a superposition state within shared weights. This coupling leads to the "memory wall," where computational capacity is squandered on simulating retrieval, often resulting in hallucinations. In this paper, we propose "digital metabolism," a thermodynamic hypothesis suggesting that targeted forgetting is necessary for distilling a pure neural logic core. To validate this hypothesis, we introduce the Regenerative Logic-Core Protocol (RLCP), a dual-stream training framework that renders specific factual dependencies linearly undecodable via deep-layer gradient reversal. Applying RLCP to Qwen2.5-0.5B, we observe a distinct phase transition: the model achieves near-zero retention of targeted factual associations (Accuracy < 7%) while exhibiting changes consistent with an emergent "structural crystallization" effect. Empirical analysis on GSM8K reveals that the "metabolized" model spontaneously adopts chain-of-thought (CoT) scaffolding, which we interpret as compensating for the loss of direct associative recall (shifting from $O(1)$ recall to $O(N)$ reasoning). While the causal mechanism underlying this behavioral shift requires further investigation, our findings provide a dynamic weight-level counterpart to architectural innovations like DeepSeek's Engram, paving the way for modular "Neural CPU + Symbolic RAM" architectures.

Digital Metabolism: Decoupling Logic from Facts via Regenerative Unlearning -- Towards a Pure Neural Logic Core

TL;DR

This work tackles parameter entanglement in LLMs, where logic and factual knowledge share weights, causing a memory wall and hallucinations. It introduces digital metabolism and the Regenerative Logic-Core Protocol (RLCP), a dual-stream training framework that uses deep-layer gradient reversal to render targeted factual dependencies linearly undecodable while preserving context-driven reasoning. In experiments on Qwen2.5-0.5B-Instruct, RLCP drives factual retention to accuracy and produces a phase transition toward a structural crystallization of representations, accompanied by emergent chain-of-thought scaffolding on GSM8K. These results suggest a viable, training-based path to decouple facts and logic, with implications for modular neural CPU plus symbolic RAM architectures and improved robustness to hallucination.

Abstract

Large language models (LLMs) currently suffer from parameter entanglement, where general reasoning capabilities (logic) and specific factual knowledge (facts) exist in a superposition state within shared weights. This coupling leads to the "memory wall," where computational capacity is squandered on simulating retrieval, often resulting in hallucinations. In this paper, we propose "digital metabolism," a thermodynamic hypothesis suggesting that targeted forgetting is necessary for distilling a pure neural logic core. To validate this hypothesis, we introduce the Regenerative Logic-Core Protocol (RLCP), a dual-stream training framework that renders specific factual dependencies linearly undecodable via deep-layer gradient reversal. Applying RLCP to Qwen2.5-0.5B, we observe a distinct phase transition: the model achieves near-zero retention of targeted factual associations (Accuracy < 7%) while exhibiting changes consistent with an emergent "structural crystallization" effect. Empirical analysis on GSM8K reveals that the "metabolized" model spontaneously adopts chain-of-thought (CoT) scaffolding, which we interpret as compensating for the loss of direct associative recall (shifting from recall to reasoning). While the causal mechanism underlying this behavioral shift requires further investigation, our findings provide a dynamic weight-level counterpart to architectural innovations like DeepSeek's Engram, paving the way for modular "Neural CPU + Symbolic RAM" architectures.
Paper Structure (25 sections, 2 theorems, 15 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 25 sections, 2 theorems, 15 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The Entanglement Dilemma
  3. The Memory Wall and Hallucination
  4. The Thermodynamic Hypothesis: Neural Recycling
  5. Related Work
  6. Parametric vs. Non-Parametric Memory
  7. Architectural Decoupling: The Soft--Hard Duality
  8. Theoretical Framework
  9. Information Bottleneck and Logic Distillation
  10. The Entanglement Field Equation
  11. Methodology: Regenerative Logic-Core Protocol
  12. Adversarial Architecture
  13. Deep-Layer Gradient Reversal
  14. Algorithm and Schedule
  15. Experiments
  16. ...and 10 more sections

Key Result

Proposition 3.2

Suppose Assumption ass:orthogonality holds with parameter $\delta$. Consider a parameter update of the form where $\eta > 0$ is the learning rate. Then the change in logical task loss satisfies In particular, when $\delta \approx 0$ (near-orthogonality), minimizing factual retention has negligible first-order impact on logical task performance.

Figures (4)

  • Figure 1: Classification Indistinguishability. t-SNE visualization of latent states at layer 20. The RLCP model's representations of cities (red) and fruits (green) are intermixed in the semantic subspace, confirming that the linear separability of factual identity has been destroyed. This represents a phase transition from "recall state" to "tabula rasa state."
  • Figure 2: Semantic Subspace Collapse. Specific facts (individual numbers) collapse into tight logic types. The model preserves abstract structure but loses linearly decodable identity binding.
  • Figure 3: Thermodynamic Cooling of Attention Mechanics (Neural Recycling). Visualization of attention weights from layer 20 heads attending to context tokens. (A) Baseline (Just-RAG): The model exhibits high-entropy attention ($H = 1.59$), with diffuse focus on functional tokens ("is," "located") and internal residual streams, indicating interference from internal memory. (B) Digital Metabolism (RLCP): Following metabolic unlearning, the model demonstrates a phase transition to low entropy ($H = 0.90$). Attention heads exhibit focused attention on the external retrieval target ("Germany"). This confirms the hypothesis that neural resources previously entangled in memory storage are recycled for precise algorithmic context processing.
  • Figure 4: Attention Distribution at Layer 20 (Prediction Step). A quantitative comparison of attention weights allocated to the evidence token. The RLCP model (red) assigns significantly higher weight (approximately 0.7) to the evidence compared to the baseline (blue, less than 0.1), demonstrating that the metabolized model is structurally forced to rely on context rather than internal priors.

Theorems & Definitions (6)

  • Proposition 3.2: Bounded Impact of Factual Unlearning on Logic
  • proof
  • Remark 3.3: Gap Between Theory and Practice
  • Corollary 3.4: Bound on Logic Loss Change under Composite Updates
  • proof
  • Remark 3.5: Empirical Validation of Orthogonality