Exposía: Academic Writing Assessment of Exposés and Peer Feedback

TL;DR

Exposía is the first public dataset that links student exposé writing, peer and instructor feedback, and revision in a higher-education setting, enabling analysis of drafting–feedback–revision workflows. The authors benchmark open-source LLMs on two tasks—exposé scoring and feedback scoring—using zero-shot prompting with both single-criterion and combined-criterion variants, finding that LLMs align well with human judgments on non-domain criteria but struggle on content- or in-domain aspects; combined prompts generally yield higher agreement and better efficiency. The results highlight the potential of LLMs as classroom decision-support tools rather than fully automated graders, with guidance on deployment strategies (e.g., batching, peak-load handling) and a rich rubric-driven evaluation framework to ensure educational relevance. Ethical considerations, consent limitations, and ongoing data collection are acknowledged, and the dataset is positioned as a foundation for broader cross-domain research into educational writing assessment and feedback analysis.

Abstract

We present Exposía, the first public dataset that connects writing and feedback assessment in higher education, enabling research on educationally grounded approaches to academic writing evaluation. Exposía includes student research project proposals and peer and instructor feedback consisting of comments and free-text reviews. The dataset was collected in the "Introduction to Scientific Work" course of the Computer Science undergraduate program that focuses on teaching academic writing skills and providing peer feedback on academic writing. Exposía reflects the multi-stage nature of the academic writing process that includes drafting, providing and receiving feedback, and revising the writing based on the feedback received. Both the project proposals and peer feedback are accompanied by human assessment scores based on a fine-grained, pedagogically-grounded schema for writing and feedback assessment that we develop. We use Exposía to benchmark state-of-the-art open-source large language models (LLMs) for two tasks: automated scoring of (1) the proposals and (2) the student reviews. The strongest LLMs attain high agreement on scoring aspects that require little domain knowledge but degrade on dimensions evaluating content, in line with human agreement values. We find that LLMs align better with the human instructors giving high scores. Finally, we establish that a prompting strategy that scores multiple aspects of the writing together is the most effective, an important finding for classroom deployment.

Figures (9)

• Figure 1: Overview of Exposía. In the university course "Introduction to Scientific Work", students submit a draft exposé, receive feedback in the form of comments and a free-text review, revise and submit a final exposé. Feedback is produced by peers and instructors. Instructors grade draft and final exposés and student feedback.
• Figure 2: Example instance from Exposía. Top: A student draft exposé (top-left) is reviewed by an instructor or a student peer and receives feedback. The feedback consists of (1) passage-anchored inline comments (shown in the top middle in red and green), each associated with a tag (e.g., Weakness, Strength); and (2) a free-text review (top-right). Bottom: Exposés and reviews are graded by instructors and receive scores on multiple aspects (criteria) organized into high-level topics (rubrics). The exposé grading criteria and the review grading criteria are shown in the bottom left and bottom right of the figure, respectively.
• Figure 3: Per-rubric score improvements from draft to final exposé. Each bar shows the mean change in score. Error bars indicate nonparametric bootstrap 95% confidence intervals for each item.
• Figure 4: Human--LLM agreement (QWA) for exposé scoring by expertise level. The dotted line shows human-human QWA.
• Figure 5: Group asymmetry in human--LLM agreement by exposé rubric. Each point shows the difference in agreement between an LLM and human raters (Group 1 (G1) vs. Group 2 (G2)). The vertical line at $\Delta=0$ indicates equal agreement of the LLM with human G1 and G2 raters. Points to the right indicate the model agrees more with G1 raters than with G2 raters.