What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Models

TL;DR

The paper tackles Echo Reflection, a failure mode where LLMs fail to generate novel insights during reflection in knowledge-intensive tasks. It introduces Adaptive Entropy Policy Optimization (AEPO), a two-component framework consisting of Reflection-aware Information Filtration (RIF) based on Information Bottleneck theory and Adaptive Entropy Optimization (AEO) to regulate information flow and balance exploration. Through token-level proxies and an IB-grounded objective, AEPO suppresses erroneous intermediate content while preserving task-relevant information and dynamically tunes reasoning entropy across stages. Empirical results on medical-domain QA and diverse OOD benchmarks show state-of-the-art performance over mainstream RL baselines, with notable gains when combined with domain-specific fine-tuning and across prompts, indicating strong generalization and practical impact for knowledge-intensive reasoning tasks.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of reasoning tasks. Recent methods have further improved LLM performance in complex mathematical reasoning. However, when extending these methods beyond the domain of mathematical reasoning to tasks involving complex domain-specific knowledge, we observe a consistent failure of LLMs to generate novel insights during the reflection stage. Instead of conducting genuine cognitive refinement, the model tends to mechanically reiterate earlier reasoning steps without introducing new information or perspectives, a phenomenon referred to as "Echo Reflection". We attribute this behavior to two key defects: (1) Uncontrollable information flow during response generation, which allows premature intermediate thoughts to propagate unchecked and distort final decisions; (2) Insufficient exploration of internal knowledge during reflection, leading to repeating earlier findings rather than generating new cognitive insights. Building on these findings, we proposed a novel reinforcement learning method termed Adaptive Entropy Policy Optimization (AEPO). Specifically, the AEPO framework consists of two major components: (1) Reflection-aware Information Filtration, which quantifies the cognitive information flow and prevents the final answer from being affected by earlier bad cognitive information; (2) Adaptive-Entropy Optimization, which dynamically balances exploration and exploitation across different reasoning stages, promoting both reflective diversity and answer correctness. Extensive experiments demonstrate that AEPO consistently achieves state-of-the-art performance over mainstream reinforcement learning baselines across diverse benchmarks.

Figures (7)

• Figure 1: Illustrative examples of: (a) limited or declined performance improvement on original RL-based methods. (b) a typical "Echo Reflection" of GRPO-based method reasoning (The echoed wrong contents are highlighted in red).
• Figure 2: The overall framework of the proposed AEPO algorithm. It consists of two key components: 1) Reflection-aware Information Filtration (RIF), which leverages Information Bottleneck theory to constrain the flow of cognitive information; (2) Adaptive Entropy Optimization (AEO), which balances the behaviors of LLM exploration and exploration.
• Figure 3: Information Bottleneck adopted in Reflection-aware Information Filtration module. The Information Bottleneck aims to suppress erroneous information and redundant information flow (left). By minimizing $\mathcal{L}_{IB}$, the token entropy is reduced (right).
• Figure 4: Visualization of performances on O.O.D. datasets.
• Figure 5: Violin plot of creativity index on MedMcQA.