Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Multiple Mini Interview (MMI) Scoring

Ryan Huynh, Frank Guerin, Alison Callwood

TL;DR

This paper tackles the challenge ofautomating MMI scoring for soft skills by showing that a two-stage multi-agent prompting framework—with a preprocessing step and nine single-criterion scoring agents using 3-shot in-context learning—outperforms state-of-the-art fine-tuning methods and approaches human reliability. The approach achieves an average Quadratic Weighted Kappa of about $0.62$ and a mean squared error around $0.87$ on MMI data, significantly surpassing baselines. It also generalises to the ASAP AES benchmark, matching or exceeding domain-specific models without additional training, suggesting that structured prompt engineering can be a scalable alternative for complex, subjective assessment tasks. The work highlights the importance of rubric abstraction, exemplar selection, and per-criterion reasoning, with implications for reliable, fair, and cost-effective automated assessments in high-stakes contexts.

Abstract

Assessing soft skills such as empathy, ethical judgment, and communication is essential in competitive selection processes, yet human scoring is often inconsistent and biased. While Large Language Models (LLMs) have improved Automated Essay Scoring (AES), we show that state-of-the-art rationale-based fine-tuning methods struggle with the abstract, context-dependent nature of Multiple Mini-Interviews (MMIs), missing the implicit signals embedded in candidate narratives. We introduce a multi-agent prompting framework that breaks down the evaluation process into transcript refinement and criterion-specific scoring. Using 3-shot in-context learning with a large instruct-tuned model, our approach outperforms specialised fine-tuned baselines (Avg QWK 0.62 vs 0.32) and achieves reliability comparable to human experts. We further demonstrate the generalisability of our framework on the ASAP benchmark, where it rivals domain-specific state-of-the-art models without additional training. These findings suggest that for complex, subjective reasoning tasks, structured prompt engineering may offer a scalable alternative to data-intensive fine-tuning, altering how LLMs can be applied to automated assessment.

Automated Multiple Mini Interview (MMI) Scoring

TL;DR

This paper tackles the challenge ofautomating MMI scoring for soft skills by showing that a two-stage multi-agent prompting framework—with a preprocessing step and nine single-criterion scoring agents using 3-shot in-context learning—outperforms state-of-the-art fine-tuning methods and approaches human reliability. The approach achieves an average Quadratic Weighted Kappa of about and a mean squared error around on MMI data, significantly surpassing baselines. It also generalises to the ASAP AES benchmark, matching or exceeding domain-specific models without additional training, suggesting that structured prompt engineering can be a scalable alternative for complex, subjective assessment tasks. The work highlights the importance of rubric abstraction, exemplar selection, and per-criterion reasoning, with implications for reliable, fair, and cost-effective automated assessments in high-stakes contexts.

Abstract

Assessing soft skills such as empathy, ethical judgment, and communication is essential in competitive selection processes, yet human scoring is often inconsistent and biased. While Large Language Models (LLMs) have improved Automated Essay Scoring (AES), we show that state-of-the-art rationale-based fine-tuning methods struggle with the abstract, context-dependent nature of Multiple Mini-Interviews (MMIs), missing the implicit signals embedded in candidate narratives. We introduce a multi-agent prompting framework that breaks down the evaluation process into transcript refinement and criterion-specific scoring. Using 3-shot in-context learning with a large instruct-tuned model, our approach outperforms specialised fine-tuned baselines (Avg QWK 0.62 vs 0.32) and achieves reliability comparable to human experts. We further demonstrate the generalisability of our framework on the ASAP benchmark, where it rivals domain-specific state-of-the-art models without additional training. These findings suggest that for complex, subjective reasoning tasks, structured prompt engineering may offer a scalable alternative to data-intensive fine-tuning, altering how LLMs can be applied to automated assessment.
Paper Structure (35 sections, 2 figures, 9 tables)

This paper contains 35 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The MMI Scoring Task and Dataset
  4. Methodology
  5. Initial Study: Sentence-Level Analysis
  6. Iterative Development of Prompting Strategies
  7. In-Context Learning
  8. Retrieval-Augmented Generation (RAG)
  9. Proposed Method: A Multi-Agent Framework
  10. Fine-Tuning vs. Prompting
  11. Baseline Fine-Tuning
  12. SOTA-Adapted Fine-Tuning (RMTS)
  13. Analysis of Fine-Tuning Failures
  14. Results and Analysis
  15. Main Results: SOTA Model Comparison
  16. ...and 20 more sections

Figures (2)

  • Figure 1: Distribution of human-assigned scores across all criteria, showing a clear skew towards the upper end.
  • Figure 2: Prediction Error Distribution (Predicted -- True) for RMTS-adapted Llama 3.1 8B and modernBERT across all nine criteria.