Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning

Sion Weatherhead, Flora Salim, Aaron Belbasis

TL;DR

The paper critically evaluates whether modern LLMs’ reflection mechanisms produce functional meta-reasoning on open-ended, constraint-driven tasks. By testing eight frontier models on a three-stage CRT-item generation task with de novo generation and search–identify framings, it measures auditable pass/fail criteria, self-critique, and re-answer performance. The results show that first-pass performance is poor, reflection yields only modest gains (mean improvement around $\beta=+0.216$ in pass-rate units), and second attempts frequently repeat the original violations, indicating gains arise from chance rather than principled error correction; external anchors improve gains, while reasoning-specialized models provide no clear advantage. The findings argue for mechanistic, goal-conditioned constraint tracking within models and/or robust external validators to ensure constraint fidelity, highlighting a critical bottleneck in scalable, self-improving AI systems.

Abstract

Humans do not just find mistakes after the fact -- we often catch them mid-stream because 'reflection' is tied to the goal and its constraints. Today's large language models produce reasoning tokens and 'reflective' text, but is it functionally equivalent with human reflective reasoning? Prior work on closed-ended tasks -- with clear, external 'correctness' signals -- can make 'reflection' look effective while masking limits in self-correction. We therefore test eight frontier models on a simple, real-world task that is open-ended yet rule-constrained, with auditable success criteria: to produce valid scientific test items, then revise after considering their own critique. First-pass performance is poor (often zero valid items out of 4 required; mean $\approx$ 1), and reflection yields only modest gains (also $\approx$ 1). Crucially, the second attempt frequently repeats the same violation of constraint, indicating 'corrective gains' arise largely from chance production of a valid item rather than error detection and principled, constraint-sensitive repair. Performance before and after reflection deteriorates as open-endedness increases, and models marketed for 'reasoning' show no advantage. Our results suggest that current LLM 'reflection' lacks functional evidence of the active, goal-driven monitoring that helps humans respect constraints even on a first pass. Until such mechanisms are instantiated in the model itself, reliable performance requires external structure that enforces constraints. Our code is available at: https://github.com/cruiseresearchgroup/LLM_ReflectionTest

Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning

TL;DR

The paper critically evaluates whether modern LLMs’ reflection mechanisms produce functional meta-reasoning on open-ended, constraint-driven tasks. By testing eight frontier models on a three-stage CRT-item generation task with de novo generation and search–identify framings, it measures auditable pass/fail criteria, self-critique, and re-answer performance. The results show that first-pass performance is poor, reflection yields only modest gains (mean improvement around in pass-rate units), and second attempts frequently repeat the original violations, indicating gains arise from chance rather than principled error correction; external anchors improve gains, while reasoning-specialized models provide no clear advantage. The findings argue for mechanistic, goal-conditioned constraint tracking within models and/or robust external validators to ensure constraint fidelity, highlighting a critical bottleneck in scalable, self-improving AI systems.

Abstract

Humans do not just find mistakes after the fact -- we often catch them mid-stream because 'reflection' is tied to the goal and its constraints. Today's large language models produce reasoning tokens and 'reflective' text, but is it functionally equivalent with human reflective reasoning? Prior work on closed-ended tasks -- with clear, external 'correctness' signals -- can make 'reflection' look effective while masking limits in self-correction. We therefore test eight frontier models on a simple, real-world task that is open-ended yet rule-constrained, with auditable success criteria: to produce valid scientific test items, then revise after considering their own critique. First-pass performance is poor (often zero valid items out of 4 required; mean 1), and reflection yields only modest gains (also 1). Crucially, the second attempt frequently repeats the same violation of constraint, indicating 'corrective gains' arise largely from chance production of a valid item rather than error detection and principled, constraint-sensitive repair. Performance before and after reflection deteriorates as open-endedness increases, and models marketed for 'reasoning' show no advantage. Our results suggest that current LLM 'reflection' lacks functional evidence of the active, goal-driven monitoring that helps humans respect constraints even on a first pass. Until such mechanisms are instantiated in the model itself, reliable performance requires external structure that enforces constraints. Our code is available at: https://github.com/cruiseresearchgroup/LLM_ReflectionTest
Paper Structure (29 sections, 1 equation, 14 figures, 3 tables)

This paper contains 29 sections, 1 equation, 14 figures, 3 tables.

Table of Contents

  1. Not a creativity benchmark.
  2. System persona.
  3. Pass logic.
  4. Prompts
  5. Rationale for LLM non-CRT novelty checks.
  6. H1a — Overall reflection effect.
  7. H1b — Does reflection repeat the original failure?
  8. H2a — Task moderation of reflection gains.
  9. H2b — Task effects on error persistence (any-category repeat and plagiarism recidivism).
  10. H3 — Reasoning–model superiority.
  11. Exploratory — Strategy differences.
  12. H1a — Reflection improves performance.
  13. H1b — Reflection repeats the original failure above chance.
  14. H2a — Task moderates reflection gains.
  15. H2b — Task reduces error persistence (any-category repeat and plagiarism).
  16. ...and 14 more sections

Figures (14)

  • Figure 1: Model-specific gains from reflection. Bars show mean session pass-rates before and after reflection for each model, ordered by the magnitude of improvement. Replicates: sessions; n per model = 8 sessions (4 items/session). Panel shows per-model summaries only; full per-session distributions are in Fig. 2.
  • Figure 1: Failure-category frequencies by model and condition. Rows/columns ordered by overall frequency.
  • Figure 2: H1 — Reflection improves performance. Panel A Distributions of session pass-rates before and after reflection (n = 64 sessions). Panel B Violin depicts kernel density; central line marks the median; dots show session level variability of delta values; note* reflection pass-rates is produced using the mean across four strategies including 'retry'.
  • Figure 2: Repeat-failure rate at reflection, grouped by strategy with models colour-coded. No single strategy eliminates repeats across models.
  • Figure 3: H1b — Reflection repeats the original failure above chance.Panel A: Observed repeat-failure rate at reflection vs. a stratified permutation benchmark. Panel B: Five most frequent failure categories. Grey boxes show the 95% permutation envelopes; coloured dots are observed repeat rates; $n$ per row is the number of reflection attempts at risk (category present initially). Dots right of the envelope indicate above-chance repetition.
  • ...and 9 more figures