The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solving

TL;DR

This work identifies a fundamental tension in large language model training: maximizing a single scalar objective tends to collapse the diversity of reasoning strategies, hindering creativity and generalization. It introduces Distributional Creative Reasoning (DCR), a variational framework that optimizes over the full distribution of solution traces and adds a Diversity Energy term $\mathcal{D}[p]=\alpha H[p]-\beta Q[p]$ to promote both breadth and semantic novelty. The authors prove a Diversity Decay Theorem showing algorithm‑specific collapse modes under scalar objectives, and they show that incorporating the DCR objective guarantees convergence to a unique, stable, and diverse interior equilibrium by shaping the policy with an entropy component and a kernel‑based diversity term focused on correct traces. They also outline a practical design space—the Creativity Kernel—for sculpting structured diversity and offer actionable recipes for tuning hyperparameters, enabling LLMs to be both correct and creative. The framework yields testable predictions and provides a principled path from theoretical insights to real‑world improvement in creative problem solving for LLMs.

Abstract

State-of-the-art large language model (LLM) pipelines rely on bootstrapped reasoning loops: sampling diverse chains of thought and reinforcing the highest-scoring ones, mainly optimizing correctness. We analyze how this design choice is sensitive to the collapse of the model's distribution over reasoning paths, slashing semantic entropy and undermining creative problem-solving. To analyze this failure, we introduce Distributional Creative Reasoning (DCR), a unified variational objective that casts training as gradient flow through probability measures on solution traces. STaR, GRPO, and DPO, as well as entropy bonuses, and other methods, all constitute special cases of the same loss. The framework delivers three core results: (i) the diversity decay theorem, describing how correctness-based objectives lead to distinct modes of diversity decay for STaR, GRPO, and DPO; (ii) designs that ensure convergence to a stable and diverse policy, effectively preventing collapse; and (iii) simple, actionable recipes to achieve this in practice. DCR thus offers the first principled recipe for LLMs that remain both correct and creative.

Figures (9)

• Figure 1: Strategy–simplex dynamics. Representative trajectories of cluster masses $(m_A,m_B,m_C)$ under STaR, GRPO, DPO, and DCR. STaR collapses to a vertex; GRPO drifts along the face; DPO equalizes on the face; DCR reaches a stable interior point retaining all clusters. Early (step 200) and late (step 5000) states are marked.
• Figure 2: Study A: collapse modes. Rows: STaR (top), GRPO (middle), DPO (bottom). Columns: entropy $H$, fixation index $\mathrm{Fix}$, cluster Gini, incorrect mass (log scale). STaR deterministically fixates; GRPO drifts with speed increasing at smaller batch; DPO equalizes among correct traces while keeping incorrect mass at 0.
• Figure 3: Theory vs. procedural overlays (single seed). Entropy and cluster–Gini trajectories for STaR, GRPO, and DPO. Procedural updates (sequential STaR, group REINFORCE, Davidson–ties DPO) track theory closely in events; instantaneous directions differ most for DPO.
• Figure 4: Alignment vs. theory over time. For each method: cosine of $\Delta p$ (solid: Euclidean; dotted: Shahshahani), sign agreement of log–ratio slopes, and event–time gap (procedural $-$ theory). DPO: low cosine, near–perfect signs; GRPO: near–neutral; STaR: high cosine, zero gap.
• Figure 5: Alignment summary vs. batch size. Euclidean/Shahshahani cosine and one–step JS divergence as functions of $B$ (markers: mean; bars: s.d.). Cosine decreases with $B$ for DPO while JS concurrently decreases, indicating increasingly synchronous trajectories despite metric/parameterization mismatch.

Theorems & Definitions (90)

• Proposition 3.1: Concavity of $\mathcal{D}$, cf. \ref{['appA:functionals']}
• Theorem 3.1: Global Convergence of DCR Training, cf. \ref{['appA:flow']}, \ref{['thm:global-convergence']}
• Theorem 4.1: Diversity Decay Theorem
• Lemma A.1: Mean–log bounds and entropic Lipschitzness
• proof
• Proposition A.1: Interior maximizers
• proof
• Lemma A.2: Bounded fitness implies interiority
• proof
• Remark A.1: Applicability