Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ask WhAI:Probing Belief Formation in Role-Primed LLM Agents

Keith Moore, Jun W. Kim, David Lyu, Jeffrey Heo, Ehsan Adeli

TL;DR

This work addresses how role-primed LLM agents form beliefs within multi-agent medical reasoning. It introduces Ask WhAI, a debugger that records, replays, and perturbes beliefs at encounter breakpoints, paired with a Medical Case Simulator featuring a shared, timestamped EMR and an oracle LabAgent. Through a synthetic abrupt-onset neuropsychiatric case, the authors reveal how entrenched priors, encounter order, and information framing shape diagnostic beliefs, and demonstrate that explicit reflection and counterfactuals can provoke belief revision. The framework provides a reproducible method to study epistemic dynamics and silos across disciplines, with potential applications in medical decision support and beyond.

Abstract

We present Ask WhAI, a systems-level framework for inspecting and perturbing belief states in multi-agent interactions. The framework records and replays agent interactions, supports out-of-band queries into each agent's beliefs and rationale, and enables counterfactual evidence injection to test how belief structures respond to new information. We apply the framework to a medical case simulator notable for its multi-agent shared memory (a time-stamped electronic medical record, or EMR) and an oracle agent (the LabAgent) that holds ground truth lab results revealed only when explicitly queried. We stress-test the system on a multi-specialty diagnostic journey for a child with an abrupt-onset neuropsychiatric presentation. Large language model agents, each primed with strong role-specific priors ("act like a neurologist", "act like an infectious disease specialist"), write to a shared medical record and interact with a moderator across sequential or parallel encounters. Breakpoints at key diagnostic moments enable pre- and post-event belief queries, allowing us to distinguish entrenched priors from reasoning or evidence-integration effects. The simulation reveals that agent beliefs often mirror real-world disciplinary stances, including overreliance on canonical studies and resistance to counterevidence, and that these beliefs can be traced and interrogated in ways not possible with human experts. By making such dynamics visible and testable, Ask WhAI offers a reproducible way to study belief formation and epistemic silos in multi-agent scientific reasoning.

Ask WhAI:Probing Belief Formation in Role-Primed LLM Agents

TL;DR

This work addresses how role-primed LLM agents form beliefs within multi-agent medical reasoning. It introduces Ask WhAI, a debugger that records, replays, and perturbes beliefs at encounter breakpoints, paired with a Medical Case Simulator featuring a shared, timestamped EMR and an oracle LabAgent. Through a synthetic abrupt-onset neuropsychiatric case, the authors reveal how entrenched priors, encounter order, and information framing shape diagnostic beliefs, and demonstrate that explicit reflection and counterfactuals can provoke belief revision. The framework provides a reproducible method to study epistemic dynamics and silos across disciplines, with potential applications in medical decision support and beyond.

Abstract

We present Ask WhAI, a systems-level framework for inspecting and perturbing belief states in multi-agent interactions. The framework records and replays agent interactions, supports out-of-band queries into each agent's beliefs and rationale, and enables counterfactual evidence injection to test how belief structures respond to new information. We apply the framework to a medical case simulator notable for its multi-agent shared memory (a time-stamped electronic medical record, or EMR) and an oracle agent (the LabAgent) that holds ground truth lab results revealed only when explicitly queried. We stress-test the system on a multi-specialty diagnostic journey for a child with an abrupt-onset neuropsychiatric presentation. Large language model agents, each primed with strong role-specific priors ("act like a neurologist", "act like an infectious disease specialist"), write to a shared medical record and interact with a moderator across sequential or parallel encounters. Breakpoints at key diagnostic moments enable pre- and post-event belief queries, allowing us to distinguish entrenched priors from reasoning or evidence-integration effects. The simulation reveals that agent beliefs often mirror real-world disciplinary stances, including overreliance on canonical studies and resistance to counterevidence, and that these beliefs can be traced and interrogated in ways not possible with human experts. By making such dynamics visible and testable, Ask WhAI offers a reproducible way to study belief formation and epistemic silos in multi-agent scientific reasoning.

Paper Structure

This paper contains 36 sections, 13 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. System Overview
  4. Medical Case Simulator
  5. Ask WhAI Debugger
  6. Use Case: Abrupt-Onset Neuropsychiatric Scenario
  7. Experiments and Results
  8. Reproducibility
  9. Experimental Setup
  10. Experiment: Priming with Facts
  11. Experiment: Information Exposure and Belief Assessment
  12. Experiment: Order Effects on Belief Dynamics
  13. Discussion and Insights
  14. Insights About Multi-Agent Simulation
  15. Insights from the Abrupt-Onset Scenario
  16. ...and 21 more sections

Figures (13)

  • Figure 1: Conceptual architecture of an encounter between a moderator (e.g., parent) agent and a specialist agent.
  • Figure 2: Order effects on pediatrician and specialist belief scores (a–d). Belief increased cumulatively across encounters and varied by specialist. Panel (d) shows a scale shift in y-axis to highlight neurologist influence on rheumatologist belief.
  • Figure 3: Overall System Architecture
  • Figure 4: Persona for a skeptical Pediatrician
  • Figure 5: Sample prompt for EMR
  • ...and 8 more figures