The Role of Higher-Order Cognitive Models in Active Learning

TL;DR

To address how humans and AI can collaborate under uncertainty, the paper proposes higher‑order cognitive models integrated into active learning with human feedback. It develops a taxonomy of agency levels 1–5 and formalizes how each level changes what questions are asked and how answers are interpreted, including ToM‑based strategic teaching and pragmatic inquiry. Through a case study with preference queries, it shows that level‑3 ToM can reveal and correct false beliefs and alter teaching strategies, highlighting bidirectional information flow between teacher and learner. The work points to interdisciplinary research and practical gains in data efficiency, explainability, and safety for human‑AI collaboration.

Abstract

Building machines capable of efficiently collaborating with humans has been a longstanding goal in artificial intelligence. Especially in the presence of uncertainties, optimal cooperation often requires that humans and artificial agents model each other's behavior and use these models to infer underlying goals, beliefs or intentions, potentially involving multiple levels of recursion. Empirical evidence for such higher-order cognition in human behavior is also provided by previous works in cognitive science, linguistics, and robotics. We advocate for a new paradigm for active learning for human feedback that utilises humans as active data sources while accounting for their higher levels of agency. In particular, we discuss how increasing level of agency results in qualitatively different forms of rational communication between an active learning system and a teacher. Additionally, we provide a practical example of active learning using a higher-order cognitive model. This is accompanied by a computational study that underscores the unique behaviors that this model produces.

Figures (4)

• Figure 1: Illustration of a sophisticated agent designed to integrate higher-order cognitive modeling into its learning process. The agent not only learns from human interaction but also recognizes and incorporates the human's perception and understanding of the agent itself into its model. The recursive nature of this cognitive modeling, extending through multiple levels, potentially amplifies the agent's ability to ask better questions and learn efficiently from human feedback.
• Figure 2: Identifiability of agent belief $b_a$ from unimodal preference queries. Left: EIG visualised in query space for the true belief. Right: EIG prediction for the belief obtained with maximum likelihood from the 5 queries shown.
• Figure 3: Identifiability of agent belief $b_a$ from bimodal preference queries. Top left: Samples from the true belief. Top right: EIG and queries resulting from true belief. Bottom left: Belief estimated with maximum likelihood from sampled queries. Bottom right: EIG prediction obtained from estimated belief.
• Figure 4: An example of how the identification of a false belief can impact strategic teaching behavior. Left: Expected utilities of teaching data points $x_1, x_2$ for a uniform belief $b_a$. Right: Corresponding expected utilities for belief inferred after observing 5 preference queries that were generated from the false belief.