Meta-Cognitive Reinforcement Learning with Self-Doubt and Recovery

TL;DR

This work tackles the problem of RL reliability under unreliable and non-stationary feedback by proposing a meta-cognitive framework that regulates learning via a hidden internal signal. A meta-trust variable, driven by Value Prediction Error Stability ($VPES$), governs when to allow learning updates and when to apply a fail-safe suppression, followed by a gradual recovery of learning capacity as stability returns. The approach integrates seamlessly with PPO and other actor-critic methods, and the key contributions include the VPES-based stability monitor, asymmetric trust dynamics, and a two-time-scale control loop. Empirical results on reward-corrupted continuous-control tasks show substantial reductions in late-stage failures and improved tail-risk metrics, demonstrating robustness and practical viability for long-horizon and safety-critical RL deployments.

Abstract

Robust reinforcement learning methods typically focus on suppressing unreliable experiences or corrupted rewards, but they lack the ability to reason about the reliability of their own learning process. As a result, such methods often either overreact to noise by becoming overly conservative or fail catastrophically when uncertainty accumulates. In this work, we propose a meta-cognitive reinforcement learning framework that enables an agent to assess, regulate, and recover its learning behavior based on internally estimated reliability signals. The proposed method introduces a meta-trust variable driven by Value Prediction Error Stability (VPES), which modulates learning dynamics via fail-safe regulation and gradual trust recovery. Experiments on continuous-control benchmarks with reward corruption demonstrate that recovery-enabled meta-cognitive control achieves higher average returns and significantly reduces late-stage training failures compared to strong robustness baselines.

Figures (4)

• Figure 1: System-level architecture of the proposed meta-cognitive reinforcement learning framework. The agent is augmented with a meta-cognitive controller that monitors internal learning stability, maintains a trust state, and regulates learning dynamics through top-down supervision.
• Figure 2: Overall training loop with asymmetric meta-cognitive control. The figure illustrates the execution flow of a single training iteration, where policy updates are regulated by meta-trust and repeated across iterations ($t \rightarrow t{+}1$).
• Figure 3: Meta-cognitive control dynamics over time in a representative run. Spikes in internal instability (VPES) are followed by reductions in meta-trust and learning-rate scale, while stabilization allows for gradual trust recovery and renewed learning.
• Figure 4: Per-seed final performance comparison between the recovery-enabled method and the Elastic-PPO baseline. While performance varies across random seeds, the recovery mechanism avoids extreme failure modes and exhibits more controlled tail behavior, illustrating heterogeneous recovery trajectories across seeds, with several runs achieving substantially higher returns.