Weight-of-Thought Reasoning: Exploring Neural Network Weights for Enhanced LLM Reasoning

TL;DR

Weight-of-Thought (WoT) reasoning shifts the focus from generating linear output chains to exploiting the internal weight structure of neural networks. By extracting reasoning pathways from weights via a Weight Analyzer $\Psi$ and routing information through a graph of specialized nodes with weight-directed message passing, WoT enables parallel, non-linear reasoning and improved interpretability. Empirical results across syllogistic, mathematical, algebraic, combinatorial, and geometric tasks show WoT achieving state-of-the-art performance with only ~2M parameters, far fewer than large CoT models, and with substantially lower latency. The work introduces a formal WoT mapping $\mathcal{F}: \mathbf{x} \rightarrow \mathbf{y}$ with embedding, pathway-guided node initialization, multi-round message passing, pathway-aware aggregation, and sequential refinement, and highlights the approach's potential to transform neural reasoning, especially under resource constraints.

Abstract

Large language models (LLMs) have demonstrated remarkable reasoning capabilities when prompted with strategies such as Chain-of-Thought (CoT). However, these approaches focus on token-level output without considering internal weight dynamics. We introduce Weight-of-Thought (WoT) reasoning, a novel approach that examines neural network weights before inference to identify reasoning pathways. Unlike existing methods, WoT explores the weight space through graph-based message passing, multi-step reasoning processes, and attention mechanisms. Our implementation creates an interconnected graph of reasoning nodes. Experiments on diverse reasoning tasks (syllogistic, mathematical, algebraic, combinatorial, and geometric) demonstrate that WoT achieves superior performance compared to traditional methods, particularly for complex problems. This approach leads to both improved performance and greater interpretability of the reasoning process, offering a promising direction for enhancing LLM reasoning capabilities.

Figures (14)

• Figure 1: Conceptual comparison: (a) Chain-of-Thought (CoT) focuses on generating a linear sequence of output steps. (b) Weight-of-Thought (WoT) analyzes internal model weights ($\Psi$) to structure reasoning as a dynamically guided graph process, enabling non-linear pathways.
• Figure 2: Condensed WoT process flow. Weight analysis ($\Psi$) yields pathway information $\mathbf{P}$, influencing node initialization, message passing, and aggregation (indicated conceptually by red dashed arrows). Standard learnable weights $\mathbf{W}_*$ operate at each stage.
• Figure 3: Performance Breakdown by Reasoning Task Category. The performance metric uses accuracy for classification tasks (Syllogism, Geometry) and a normalized score $1/(1+\text{MSE})$ for regression tasks (Math Seq., Algebra, Combin.), so that higher values indicate better performance. WoT consistently achieves the highest scores.
• Figure 4: Multi-dimensional model comparison using a radar chart. Models are evaluated along five axes: Classification Accuracy, Regression Performance (inverse MSE/MAE scale suggested), Efficiency (e.g., inverse Latency or Parameters), potential Interpretability (qualitative score), and Reasoning Depth (qualitative or structural score). Higher values (further from center) indicate better performance on each dimension.
• Figure 5: Training convergence in terms of validation accuracy for WoT and baseline methods.