Self-Verifying Reflection Helps Transformers with CoT Reasoning

TL;DR

The paper investigates how self-verifying reflection aids multi-step chain-of-thought reasoning in transformers using a minimal non-language framework. It introduces Reflective MTP (RMTP) and Reflective Trace-Back Search (RTBS) and proves that reflection improves accuracy when verification errors are bounded, with RTBS providing extra gains under certain conditions. Empirically, tiny transformers (1M, 4M, 16M params) solving integer multiplication and Sudoku benefit from self-verification during training and execution, with RL (GRPO) boosting in-distribution performance but predominantly exploiting shallow patterns rather than achieving broad generalization. The work suggests that integrating generative reasoning with discriminative verification helps CoT reasoning across scales, while RL’s synergy with reflection is limited in its ability to transfer to more general problem-solving.

Abstract

Advanced large language models (LLMs) frequently reflect in reasoning chain-of-thoughts (CoTs), where they self-verify the correctness of current solutions and explore alternatives. However, given recent findings that LLMs detect limited errors in CoTs, how reflection contributes to empirical improvements remains unclear. To analyze this issue, in this paper, we present a minimalistic reasoning framework to support basic self-verifying reflection for small transformers without natural language, which ensures analytic clarity and reduces the cost of comprehensive experiments. Theoretically, we prove that self-verifying reflection guarantees improvements if verification errors are properly bounded. Experimentally, we show that tiny transformers, with only a few million parameters, benefit from self-verification in both training and reflective execution, reaching remarkable LLM-level performance in integer multiplication and Sudoku. Similar to LLM results, we find that reinforcement learning (RL) improves in-distribution performance and incentivizes frequent reflection for tiny transformers, yet RL mainly optimizes shallow statistical patterns without faithfully reducing verification errors. In conclusion, integrating generative transformers with discriminative verification inherently facilitates CoT reasoning, regardless of scaling and natural language.

Figures (13)

• Figure 1: The illustration of MTP, where the transformer model $\pi$ reasons the answer $A$ of a query $Q$ through $T-1$ intermediate steps.
• Figure 2: Example reasoning steps for multiplication and Sudoku, where the core planning is presented in the reasoning step ${R}_{t+1}$.
• Figure 3: Reflective reasoning based on MTP. "$\checkmark$" and "$\times$" are self-verification labels for positive and negative steps, respectively. The steps that are instantly verified as negative are highlighted in red. In RTBS, the dashed-line arrows back-propagate the negative labels, causing parental steps to be recursively rejected (orange). The green shows the steps that successfully lead to the answer.
• Figure 4: The training workflow for transformers to perform CoT reasoning.
• Figure 5: The diagram of state transitions starting from scale $n$ in the simplified reasoning, where probabilities are attached to solid lines. In (b) reflective reasoning, the dashed-line arrow presents the trace-back move after $m$ attempts in RTBS.

Theorems & Definitions (18)

• Example 1: An MTP for integer multiplication
• Example 2: An MTP for Sudoku chiTechniquesSolvingSudoku2013
• Theorem 1
• Proposition 1: RMTP Reasoning Length
• Proposition 2
• proof
• Proposition 3: RMTP Validity conditions
• Proposition 4: RTBS Validity Condition
• Lemma 1
• proof