Adapting Like Humans: A Metacognitive Agent with Test-time Reasoning

TL;DR

This paper addresses the brittleness of vision–language models when faced with novel tasks at test time. It proposes MetaCognitive Test-Time Reasoning (MCTR), a dual-module framework with a meta-reasoning memory builder and an action-reasoning policy that are guided by an adaptive scheduler and online reinforcement learning. Through extensive Atari experiments, MCTR achieves strong generalization to unseen games (9/12 top-1), outperforming supervised fine-tuning baselines and illustrating the value of metacognitive self-updating for open-ended tasks. Analyses show meta-reasoning evolves from exploratory hypotheses to goal-directed strategies and that MCTR benefits from the synergy between reflective knowledge formation and policy adaptation.

Abstract

Recent Vision-Language Models (VLMs) exhibit strong perceptual reasoning abilities, yet they often struggle to adapt efficiently when encountering novel tasks at test time. In contrast, humans leverage the metacognitive model with memory, enabling continuous strategy refinement through metacognitive control when faced with new challenges. To bridge this gap, we propose metacognitive test-time reasoning (MCTR), a framework that equips models with the ability to learn, adapt, and improve during test time through metacognitive self-updating. Inspired by the dual structure of human metacognition, MCTR comprises meta-level and object-level VLM reasoning modules, each equipped with dedicated memory systems for hierarchical adaptive reasoning. Specifically, MCTR consists of (1) a meta-reasoning module which incrementally builds a structured memory by discovering and storing task-relevant rules, environmental patterns, and action-outcome relationships from test-time observations as natural language descriptions; and (2) an action-reasoning module that determines optimal actions through context-aware perception and strategic reasoning by dynamically retrieving and integrating knowledge from memory. The action-reasoning module continuously updates its policy through proposed metacognitive test-time reinforcement learning, adapting as knowledge memory evolves. We evaluate MCTR on 45 Atari games (33 seen, 12 unseen). MCTR demonstrates robust test-time adaptation, achieving 9/12 top-1 results on unseen games compared with baselines. Analyses through ablations, learning dynamics, and case studies reveal the complementary contributions of both components and show meta-reasoning evolving toward human-like adaptation strategies.

Figures (6)

• Figure 1: Comparison between a human metacognitive model and our MCTR framework. Left: A human metacognition model nelson1990metamemoryRivers2021MetacognitionFleur2021 comprises meta-level and object-level processes with dedicated memory systems, enabling bidirectional knowledge and control flow. Brain regions associated with knowledge (orange) and control (green) processes are shown. Right: MCTR attains human-like adaptation to novel tasks without prior knowledge through two synergistic modules: a meta-reasoning module (orange) that discovers operational knowledge under dynamic scheduling, and an action-reasoning module (green) that leverages this knowledge for multi-step reasoning while adapting via test-time reinforcement learning.
• Figure 2: Overview of Meta-Cognitive Test-Time Reasoning (MCTR) Framework. The meta-reasoning module (yellow, meta-level process in human metacognitive model) performs retrospective analysis on accumulated trajectory memory and generates memory operations (<add>, <delete>, <keep>) to update knowledge memory. A scheduler dynamically adjusts meta-reasoning frequency to prioritize frequent analysis when knowledge is sparse and reduce invocations as memory matures. The meta-reasoning module (green, object-level process) handles real-time decision-making by injecting knowledge into context to guide multi-step VLM reasoning. The ARM continuously refines its policy at test time through reinforcement learning using self-supervised signals from action majority voting and real-time knowledge and trajectory memory.
• Figure 2: Ablation study on adaptive interval scheduling. Comparison of different initial intervals ($k$) and growth rates ($\lambda$) across a subset of Atari games. Bold values indicate best performance per game.
• Figure 3: MCT-RL dynamics analysis.Left: Majority voting ratio increases as the agent progressively favors higher-quality, self-consistent actions. Right: Agreement ratio with historical trajectory memory declines, indicating active policy revision with newly acquired environmental knowledge.
• Figure 4: Case study illustrating the temporal evolution of meta-reasoning knowledge across three unseen games. Early reflections exhibit exploratory metacognition focused on hypothesis generation, while later reasoning crystallises into goal-directed, procedural strategies grounded in learnt environmental dynamics.