CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring

TL;DR

CAFES tackles multimodal AES by introducing a collaborative three-agent framework that decomposes scoring into Initial Scorer, Feedback Pool Manager, and Reflective Scorer, guided by a teacher-student MLLM paradigm. The approach generates trait-level scores, trait-specific positive feedback, and iterative revisions to improve alignment with human judgments. On the EssayJudge dataset, CAFES yields an average QWK improvement of $21\%$ (absolute $+0.07$, from $0.29$ to $0.36$), with notable gains in grammatical accuracy and lexical diversity across diverse MLLMs. This work demonstrates a robust, human-aligned, multimodal AES paradigm and highlights the potential of structured multi-agent LLM systems in education.

Abstract

Automated Essay Scoring (AES) is crucial for modern education, particularly with the increasing prevalence of multimodal assessments. However, traditional AES methods struggle with evaluation generalizability and multimodal perception, while even recent Multimodal Large Language Model (MLLM)-based approaches can produce hallucinated justifications and scores misaligned with human judgment. To address the limitations, we introduce CAFES, the first collaborative multi-agent framework specifically designed for AES. It orchestrates three specialized agents: an Initial Scorer for rapid, trait-specific evaluations; a Feedback Pool Manager to aggregate detailed, evidence-grounded strengths; and a Reflective Scorer that iteratively refines scores based on this feedback to enhance human alignment. Extensive experiments, using state-of-the-art MLLMs, achieve an average relative improvement of 21% in Quadratic Weighted Kappa (QWK) against ground truth, especially for grammatical and lexical diversity. Our proposed CAFES framework paves the way for an intelligent multimodal AES system. The code will be available upon acceptance.

Figures (25)

• Figure 1: Comparisons among the traditional AES method (a), MLLM-based method (b), and our proposed multi-agent CAFES framework (c) on AES task.
• Figure 2: The framework of our proposed CAFES. The system follows a three-stage process: ❶ Initial scoring via the student MLLM; ❷ Feedback generation for each trait via the teacher MLLM; and ❸ Final reflective scoring with justification-based revision via the teacher MLLM.
• Figure 3: Reflective scorer's JSON output format.
• Figure 3: Comparison between previous AES datasets.
• Figure 4: Trait-level score improvements after reflection via CAFES across different student MLLMs.