Automatic Feedback Generation for Short Answer Questions using Answer Diagnostic Graphs

TL;DR

This work introduces Answer Diagnostic Graphs (ADG) to generate personalized feedback for short-answer reading comprehension by linking the text's logical structure to feedback templates. The system maps student responses to ADG nodes via justification cues and uses edge relations to select targeted feedback templates, aiming to improve comprehension and motivation. An empirical study with Japanese high school students shows that system-generated feedback enhances students' error recognition and engagement, though it does not significantly boost scores and requires better alignment with text structure. The approach offers a scalable pathway to provide structured, criterion-aligned feedback in reading tasks, with clear directions for future refinement and evaluation in educational settings.

Abstract

Short-reading comprehension questions help students understand text structure but lack effective feedback. Students struggle to identify and correct errors, while manual feedback creation is labor-intensive. This highlights the need for automated feedback linking responses to a scoring rubric for deeper comprehension. Despite advances in Natural Language Processing (NLP), research has focused on automatic grading, with limited work on feedback generation. To address this, we propose a system that generates feedback for student responses. Our contributions are twofold. First, we introduce the first system for feedback on short-answer reading comprehension. These answers are derived from the text, requiring structural understanding. We propose an "answer diagnosis graph," integrating the text's logical structure with feedback templates. Using this graph and NLP techniques, we estimate students' comprehension and generate targeted feedback. Second, we evaluate our feedback through an experiment with Japanese high school students (n=39). They answered two 70-80 word questions and were divided into two groups with minimal academic differences. One received a model answer, the other system-generated feedback. Both re-answered the questions, and we compared score changes. A questionnaire assessed perceptions and motivation. Results showed no significant score improvement between groups, but system-generated feedback helped students identify errors and key points in the text. It also significantly increased motivation. However, further refinement is needed to enhance text structure understanding.

Figures (4)

• Figure 1: An example of a prompt, analytic criteria, and a student response excerpted from the RIKEN SAS Dataset. Highlighted parts of the response indicate justification cues for each analytic score. We have omitted the prompt text in this example due to space limitations.
• Figure 2: Overview of Feedback Generation.
• Figure 3: An example of Answer an answer diagnosis graph. We created an ADG for each prompt text. A1, B, and C represent analytic criteria for the prompt.
• Figure 4: Overall workflow of the experiment.