Learning to Trust Experience: A Monitor-Trust-Regulator Framework for Learning under Unobservable Feedback Reliability

TL;DR

This work addresses the gap between optimization robustness and epistemic identifiability when feedback reliability is latent and the data-generating process is closed-loop. It introduces the Monitor–Trust–Regulator (MTR) framework as a modular, introspective control loop that leverages endogenous learning dynamics to infer experience credibility and softly regulate learning updates through a slow, revisable trust signal. The authors instantiate this with a self-diagnosis mechanism and demonstrate that it improves epistemic identifiability in reinforcement learning under corrupted rewards and reveals a dissociation between performance recovery and epistemic recovery in supervised learning. The framework provides a practical design principle for intrinsic reliability assessment in EIUR-like settings, with implications for AI safety, auditing, and future research into more general monitors, regulators, and larger-scale models.

Abstract

Learning under unobservable feedback reliability poses a distinct challenge beyond optimization robustness: a system must decide whether to learn from an experience, not only how to learn stably. We study this setting as Epistemic Identifiability under Unobservable Reliability (EIUR), where each experience has a latent credibility, reliable and unreliable feedback can be locally indistinguishable, and data are generated in a closed loop by the learner's own evolving beliefs and actions. In EIUR, standard robust learning can converge stably yet form high-confidence, systematically wrong beliefs. We propose metacognitive regulation as a practical response: a second, introspective control loop that infers experience credibility from endogenous evidence in the learner's internal dynamics. We formalize this as a modular Monitor-Trust-Regulator (MTR) decomposition and instantiate it with self-diagnosis, which maintains a slowly varying experience-trust variable that softly modulates learning updates, without exogenous reliability labels or an explicit corruption model. Empirically, in the EIUR regimes studied here, self-diagnosis is associated with improved epistemic identifiability. In reinforcement learning, it enables calibrated skepticism and recovery under systematically corrupted rewards. In supervised learning, it exposes a critical dissociation: performance recovery does not imply epistemic recovery. Accuracy can rebound while internal belief dynamics remain locked-in by early misleading data, a failure detectable only through introspective diagnostics. Together, MTR and self-diagnosis provide an organizing abstraction and a concrete design template for intrinsic reliability assessment in autonomous learning under unobservable reliability.

Figures (7)

• Figure 1: The Monitor--Trust--Regulator (MTR) framework for metacognitive regulation. A separable regulatory loop (top) operates alongside the primary learning loop (bottom). The Monitor $\mathcal{M}$ extracts stability descriptors from internal learning dynamics. The Trust Estimator $\mathcal{T}$ aggregates these descriptors into a slow-varying assessment of experience credibility. The Regulator $\mathcal{R}$ uses this assessment to softly modulate the influence of experience on the base learner $\mathcal{B}$. This introspective loop mitigates epistemic lock-in and improves identifiability in the EIUR regimes studied, without requiring exogenous reliability labels.
• Figure 2: Belief evolution under unreliable experience without reinforcement learning. When experience reliability is unobservable and observations are locally indistinguishable, belief formation is not identifiable without access to temporal structure in learning dynamics. Indiscriminate belief updating leads to persistent instability (blue), whereas self-diagnosis based on belief dynamics enables stable convergence toward the true latent value (orange).
• Figure 3: Mean experience trust under sustained reward corruption. Trust decreases during the corruption phase but stabilizes at a suppressed level rather than collapsing. When reliable feedback is restored, trust recovers gradually, demonstrating calibrated skepticism rather than irreversible filtering.
• Figure 4: Mean experience trust under ambiguous but unbiased reward noise. Trust forms gradually and stabilizes at moderate levels despite high-variance feedback, indicating that the system distinguishes informative but uncertain experience from systematically misleading experience.
• Figure 5: Mean experience trust under early misleading rewards. Trust is initially reduced due to systematically incorrect feedback, but consistently recovers once reliable rewards are restored, indicating that early epistemic errors are revisable rather than permanently fixed.

Theorems & Definitions (1)

• Proposition 1: Epistemic Identifiability Gap under EIUR