Reasoning Models Generate Societies of Thought

TL;DR

The paper demonstrates that advanced reasoning models organize internal thought as a 'society of thought'—a structured, multi-voice dialogue with diverse personalities and expertise. This social-like organization correlates with higher reasoning accuracy, is measurable via conversational behaviours and socio-emotional roles, and can be causally enhanced through steering of specific internal features. Reinforcement learning experiments show that conversational scaffolding accelerates the emergence of reasoning strategies and enables cross-domain transfer, suggesting a general principle: diversity and structured interaction among internal voices accelerates problem solving. Together, these findings point to a computational parallel to human collective intelligence and motivate new architectural and training strategies that harness internal diversity to improve AI reasoning and generalization.

Abstract

Large language models have achieved remarkable capabilities across domains, yet mechanisms underlying sophisticated reasoning remain elusive. Recent reasoning models outperform comparable instruction-tuned models on complex cognitive tasks, attributed to extended computation through longer chains of thought. Here we show that enhanced reasoning emerges not from extended computation alone, but from simulating multi-agent-like interactions -- a society of thought -- which enables diversification and debate among internal cognitive perspectives characterized by distinct personality traits and domain expertise. Through quantitative analysis and mechanistic interpretability methods applied to reasoning traces, we find that reasoning models like DeepSeek-R1 and QwQ-32B exhibit much greater perspective diversity than instruction-tuned models, activating broader conflict between heterogeneous personality- and expertise-related features during reasoning. This multi-agent structure manifests in conversational behaviors, including question-answering, perspective shifts, and the reconciliation of conflicting views, and in socio-emotional roles that characterize sharp back-and-forth conversations, together accounting for the accuracy advantage in reasoning tasks. Controlled reinforcement learning experiments reveal that base models increase conversational behaviors when rewarded solely for reasoning accuracy, and fine-tuning models with conversational scaffolding accelerates reasoning improvement over base models. These findings indicate that the social organization of thought enables effective exploration of solution spaces. We suggest that reasoning models establish a computational parallel to collective intelligence in human groups, where diversity enables superior problem-solving when systematically structured, which suggests new opportunities for agent organization to harness the wisdom of crowds.

Figures (13)

• Figure 1: Conversational behaviours and Bales' socio-emotional roles in chain-of-thought reasoning.a, Proportion of reasoning traces containing each conversational behaviour (question answering, perspective shift, conflict of perspectives, and reconciliation). b, Proportion of Bales' twelve socio-emotional roles expressed in reasoning traces, grouped into four higher-level categories: ask versus give information, and positive versus negative emotional roles (see \ref{['fig:edfig3']} for definitions of all twelve roles). c, Jaccard index measuring the balance of each socio-emotional role pair, defined as the number of reasoning traces containing both roles divided by the number containing either role (i.e., ask & give; positive & negative). d, Distribution of the number of distinct perspectives in reasoning traces, identified using an LLM-as-judge. e, Differences in problem complexity by the presence of conversational behaviours and higher-level socio-emotional roles in DeepSeek-R1, measured on a seven-point Likert scale (1 = extremely easy; 7 = extremely difficult) using an LLM-as-judge. Points indicate mean complexity for traces where the behaviour or role is present (red) or absent (blue). f, Differences in problem complexity by the presence of conversational behaviours and socio-emotional roles in DeepSeek-R1, measured by instruction-tuned (non-reasoning) models' error rates on the same problems (see \ref{['sec:measurements']}). Error bars indicate 95% confidence intervals.
• Figure 1: Distribution of reasoning trace length.a, Kernel density plot showing the distribution of reasoning trace length, measured by the number of words per reasoning trace. b, Kernel density plot showing the distribution of log-transformed reasoning trace length.
• Figure 2: Steering conversational features improves reasoning.a, Illustration of sparse autoencoder feature 30939 in DeepSeek-R1-Llama-8B, summarized as a discourse marker for surprise, realization, or acknowledgment in conversational settings. Conversation ratio indicates the proportion of conversational contexts among all contexts in which this feature is activated. Percentile indicates where this feature’s conversation ratio ranks among all features ($N = 32{,}768$). Sparsity refers to the fraction of tokens on which this feature activates across the entire corpus. Activation strength shows the magnitude of activation in the top-activating examples. The examples illustrate this feature’s activation within conversational turn-taking contexts. b, Results of a steering experiment using the activation-addition method. Adding the feature 30939 vector with a strength of 10 doubles accuracy on a complex counting task. The inset shows the causal change in conversational behaviours induced by steering this feature. c, Violin plots showing accuracy improvements from steering feature 30939, compared with a randomly selected conversational SAE feature and a randomly selected non-conversational SAE feature. d, Cognitive behaviours—including verification, backtracking, subgoal setting, and backward chaining—are causally associated with steering the activation of feature 30939. e, Structural equation model results showing that steering feature 30939 from 0 to $+10$ has both a direct effect on reasoning accuracy and a significant indirect effect mediated through cognitive behaviours (verification, subgoal setting, and backward chaining). Bold coefficients indicate statistical significance ($\emph{p} < 0.05$). ***$\emph{p} < 0.001$, **$\emph{p} < 0.01$, *$\emph{p} < 0.05$.
• Figure 2: Conversation excerpt in DeepSeek-R1 reasoning traces.a, Representative excerpt from a chemistry problem-solving trace showing multi-turn dialogue between distinct cognitive personas. Each utterance is annotated with conversational behaviors (blue) and socio-emotional roles (yellow). b, Big Five personality profiles for the five personas identified via LLM-as-judge. Radar charts display normalized trait scores (1--5 scale) for Extraversion (E), Agreeableness (A), Conscientiousness (C), Neuroticism (N), and Openness (O). Each persona exhibits domain expertise profiles. For detailed coding procedures and additional annotated examples, see \ref{['sec:annotations']}.
• Figure 3: Personality and expertise diversity in reasoning traces.a, Personality diversity of implicit reasoning perspectives inferred from each reasoning trace using an LLM-as-judge and the BFI-10 (10-Item Big Five Personality Inventory). For each Big Five dimension, diversity is quantified as the standard deviation across inferred personalities. Reasoning models (DeepSeek-R1 and QwQ-32B) exhibit markedly higher diversity in openness, neuroticism, agreeableness, and extraversion. Kernel density estimation (KDE) plots show the distribution of personality traits across reasoning traces. b, Embedding space of expertise identified by the LLM-as-judge, projected into two dimensions using UMAP and rendered with an energy-minimization layout, revealing coherent and consistent skill proximities. c, Expertise diversity of implicit reasoning perspectives inferred from each reasoning trace, measured as the mean cosine distance between each expertise-related embedding and the centroid of all embeddings in the semantic space. Reasoning models exhibit substantially greater expertise diversity than non-reasoning models. d, Sparse autoencoder (SAE) schema and feature identification underlying the steering experiments. e, Design of the steering experiment. SAE feature 30939—capturing a discourse marker for surprise, realization, or acknowledgment indicative of persona and perspective shifts—is increased or decreased with a steering strength of 10. Example reasoning traces illustrate that negative steering induces linear chain-of-thought trajectories, no steering yields subtle perspective shifts enabling self-checking, and positive steering induces frequent and pronounced perspective shifts that explore fundamentally different solution strategies. f, g, Distributions of coverage and entropy for SAE personality-related (f) and expertise-related (g) features under feature 30939 steering. Error bars indicate 95% confidence intervals; solid horizontal lines denote medians and dashed lines indicate interquartile ranges (25th–75th percentiles).