Hierarchical Pedagogical Oversight: A Multi-Agent Adversarial Framework for Reliable AI Tutoring

TL;DR

The paper tackles reliability gaps in AI tutors, where LLMs can validate incorrect student reasoning or over-simplify answers. It proposes Hierarchical Pedagogical Oversight (HPO), a three-phase framework that grounds dialogue context through Intelligence Distillation, enforces a structured adversarial five-act debate, and synthesizes judgments to produce robust classifications. On MRBench, an 8B-parameter HPO-FT model achieves Macro F1 = 0.845, outperforming GPT-4o (0.812) while using far fewer parameters, illustrating that adversarial structure can surpass mere scaling in specialized educational tasks. The work highlights the importance of grounding and moderator-driven reasoning for safe, low-resource pedagogical oversight, while noting limitations in cross-domain generalization and latency that warrant future deployment and validation efforts.

Abstract

Large Language Models (LLMs) are increasingly deployed as automated tutors to address educator shortages; however, they often fail at pedagogical reasoning, frequently validating incorrect student solutions (sycophancy) or providing overly direct answers that hinder learning. We introduce Hierarchical Pedagogical Oversight (HPO), a framework that adapts structured adversarial synthesis to educational assessment. Unlike cooperative multi-agent systems that often drift toward superficial consensus, HPO enforces a dialectical separation of concerns: specialist agents first distill dialogue context, which then grounds a moderated, five-act debate between opposing pedagogical critics. We evaluate this framework on the MRBench dataset of 1,214 middle-school mathematics dialogues. Our 8B-parameter model achieves a Macro F1 of 0.845, outperforming GPT-4o (0.812) by 3.3% while using 20 times fewer parameters. These results establish adversarial reasoning as a critical mechanism for deploying reliable, low-compute pedagogical oversight in resource-constrained environments.