Automated Formative Feedback for Short-form Writing: An LLM-Driven Approach and Adoption Analysis
Tiago Fernandes Tavares, Luciano Pereira Soares
TL;DR
Problem addressed: delivering timely formative feedback for short, biweekly engineering Capstone reports using LLMs. Approach: design and evaluate an LLM-powered feedback tool that analyzes drafts, maps tasks to evidences, and presents iterative, non-prescriptive guidance; tested across two rounds with 76 students in a Brazilian program. Key findings: adoption was initially low, but students who used the tool improved the completeness and quality of their reports, and the system's task parsing offered new organizational insights. Implications: AI-driven formative feedback can support students and advisors, though uptake and context matter, and the framework could extend to other writing tasks.
Abstract
This paper explores the development and adoption of AI-based formative feedback in the context of biweekly reports in an engineering Capstone program. Each student is required to write a short report detailing their individual accomplishments over the past two weeks, which is then assessed by their advising professor. An LLM-powered tool was developed to provide students with personalized feedback on their draft reports, guiding them toward improved completeness and quality. Usage data across two rounds revealed an initial barrier to adoption, with low engagement rates. However, students who engaged in the AI feedback system demonstrated the ability to use it effectively, leading to improvements in the completeness and quality of their reports. Furthermore, the tool's task-parsing capabilities provided a novel approach to identify potential student organizational tasks and deliverables. The findings suggest initial skepticism toward the tool with a limited adoption within the studied context, however, they also highlight the potential for AI-driven tools to provide students and professors valuable insights and formative support.