Why LLMs Cannot Think and How to Fix It

TL;DR

The paper tackles the question of whether LLMs can truly think by formalizing a notion of thought and showing that current decoder-only, text-space training cannot realize genuine internal deliberation. It argues that two design choices—deterministic model state and population-level training—prevent thoughts from collapsing into instance-specific decisions within the feature space, offering a theoretical framework built around a conditional mutual information formulation $I(State_t; Y_t | X_{1, dots,t})$. The authors propose architectural and training changes that inject non-determinism and align internal decisions with conversation instances, suggesting randomness injections and targeted fine-tuning (e.g., ORPO-like strategies) to enable feature-space thought processes. They discuss potential benefits, including improved reasoning and planning, reduced symmetry-related inefficiencies, and possible latency gains through speculative decoding, while acknowledging limitations such as depth-bound reasoning and the need for recurrence or parallelism. This work lays groundwork for empirical validation and guides future research toward enabling true internal deliberation in LLMs with practical implications for multimodal and embodied AI.

Abstract

This paper elucidates that current state-of-the-art Large Language Models (LLMs) are fundamentally incapable of making decisions or developing "thoughts" within the feature space due to their architectural constraints. We establish a definition of "thought" that encompasses traditional understandings of that term and adapt it for application to LLMs. We demonstrate that the architectural design and language modeling training methodology of contemporary LLMs inherently preclude them from engaging in genuine thought processes. Our primary focus is on this theoretical realization rather than practical insights derived from experimental data. Finally, we propose solutions to enable thought processes within the feature space and discuss the broader implications of these architectural modifications.

Figures (2)

• Figure 1: An example of a decoder-only transformer-based language model processing a conversation where there are two different approaches to solve a problem. Both approaches have a significant probabilistic weight. Until the model announces its selected approach through random sampling of output tokens, it cannot utilize its feature tensors to work on one specific solution approach, as it is yet undetermined (orange section of the model state). It can only work on multiple approaches simultaneously or risk the less prepared approach to be selected in the sampling process. Only the green section of the model state is available to work on the selected approach specifically. The attention across the gap in the middle is not shown for simplicity.
• Figure 2: Three sampled continuations of a conversation with ChatGPT-4 with the assistant claiming to have made a decision in the common conversation stem, but the branched continuations showing the claim to be false. The conversation demonstrates a lack of robust in-state decision making that using a fixed RNG seed for sampling wouldn't alleviate.