Scalable and Personalized Oral Assessments Using Voice AI

Abstract

Large language models have broken take-home exams. Students generate polished work they cannot explain under follow-up questioning. Oral examinations are a natural countermeasure -- they require real-time reasoning and cannot be outsourced to an LLM -- but they have never scaled. Voice AI changes this. We describe a system that conducted 36 oral examinations for an undergraduate AI/ML course at a total cost of \$15 (\$0.42 per student), low enough to attach oral comprehension checks to every assignment rather than reserving them for high-stakes finals. Because the LLM generates questions dynamically from a rubric, the entire examination structure can be shared in advance: practice is learning, and there is no exam to leak. A multi-agent architecture decomposes each examination into structured phases, and a council of three LLM families grades each transcript through a deliberation round in which models revise scores after reviewing peer evidence, achieving inter-rater reliability (Krippendorff's $α$ = 0.86) above conventional thresholds. But the system also broke in instructive ways: the agent stacked questions despite explicit prohibitions, could not randomize case selection, and a cloned professorial voice was perceived as aggressive rather than familiar. The recurring lesson is that behavioral constraints on LLMs must be enforced through architecture, not prompting alone. Students largely agreed the format tested genuine understanding (70%), yet found it more stressful than written exams (83%) -- unsurprising given that 83% had never taken any oral examination. We document the full design, failure modes, and student experience, and include all prompts as appendices.

Figures (5)

• Figure 1: End-to-end system architecture. Left: The examination pipeline decomposes each oral exam into three agent phases, with per-student context injected via dynamic variables. Right: The grading pipeline scores each transcript through two rounds of multi-model assessment followed by chair synthesis. Cases flagged for high disagreement are routed to human audit.
• Figure 2: Dimension-level grading agreement after deliberation (180 assessments). Only 2 of 180 dimension-level grades showed disagreement of 2 or more points.
• Figure 3: Mean scores by examination topic. The Experimentation gap (1.94 vs. 3.39 for Problem Framing) reflected insufficient class coverage of A/B testing, a teaching deficiency that written exams had not revealed.
• Figure 4: Examination duration vs. overall score. The shortest exam (9 min) received the highest score (19/20). Duration reflects articulation difficulty, not depth of understanding.
• Figure 5: Case selection over time. Despite instructions to "randomly select" a case, the agent chose Zillow 86% of the time in December 12--17. Removing Zillow caused it to fixate on Predictive Policing (64% of exams December 19--20). LLMs cannot randomize.