Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course

Pavlin G. Poličar, Martin Špendl, Tomaž Curk, Blaž Zupan

TL;DR

This study evaluates automated grading of written bioinformatics assignments using six large language models in a real classroom. By using a structured prompt design that combines a grading rubric with TA-graded examples, the authors compare LLM-based graders to human TAs in a blind, randomized setup and assess both scoring accuracy and feedback quality. Results show that, aside from a poorly performing small model (Llama-8B), LLMs achieve criterion-level accuracy comparable to TAs, with open-source models matching commercial ones when properly configured. The work provides concrete guidelines for implementing LLM-based grading, demonstrating feasibility, privacy advantages, and practical considerations such as rubric design, examples, and the option for manual review. Overall, the findings support deploying LLM-based grading in large courses to reduce staff workload while preserving feedback quality and student satisfaction.

Abstract

Providing students with individualized feedback through assignments is a cornerstone of education that supports their learning and development. Studies have shown that timely, high-quality feedback plays a critical role in improving learning outcomes. However, providing personalized feedback on a large scale in classes with large numbers of students is often impractical due to the significant time and effort required. Recent advances in natural language processing and large language models (LLMs) offer a promising solution by enabling the efficient delivery of personalized feedback. These technologies can reduce the workload of course staff while improving student satisfaction and learning outcomes. Their successful implementation, however, requires thorough evaluation and validation in real classrooms. We present the results of a practical evaluation of LLM-based graders for written assignments in the 2024/25 iteration of the Introduction to Bioinformatics course at the University of Ljubljana. Over the course of the semester, more than 100 students answered 36 text-based questions, most of which were automatically graded using LLMs. In a blind study, students received feedback from both LLMs and human teaching assistants without knowing the source, and later rated the quality of the feedback. We conducted a systematic evaluation of six commercial and open-source LLMs and compared their grading performance with human teaching assistants. Our results show that with well-designed prompts, LLMs can achieve grading accuracy and feedback quality comparable to human graders. Our results also suggest that open-source LLMs perform as well as commercial LLMs, allowing schools to implement their own grading systems while maintaining privacy.

Automated Assignment Grading with Large Language Models: Insights From a Bioinformatics Course

TL;DR

This study evaluates automated grading of written bioinformatics assignments using six large language models in a real classroom. By using a structured prompt design that combines a grading rubric with TA-graded examples, the authors compare LLM-based graders to human TAs in a blind, randomized setup and assess both scoring accuracy and feedback quality. Results show that, aside from a poorly performing small model (Llama-8B), LLMs achieve criterion-level accuracy comparable to TAs, with open-source models matching commercial ones when properly configured. The work provides concrete guidelines for implementing LLM-based grading, demonstrating feasibility, privacy advantages, and practical considerations such as rubric design, examples, and the option for manual review. Overall, the findings support deploying LLM-based grading in large courses to reduce staff workload while preserving feedback quality and student satisfaction.

Abstract

Providing students with individualized feedback through assignments is a cornerstone of education that supports their learning and development. Studies have shown that timely, high-quality feedback plays a critical role in improving learning outcomes. However, providing personalized feedback on a large scale in classes with large numbers of students is often impractical due to the significant time and effort required. Recent advances in natural language processing and large language models (LLMs) offer a promising solution by enabling the efficient delivery of personalized feedback. These technologies can reduce the workload of course staff while improving student satisfaction and learning outcomes. Their successful implementation, however, requires thorough evaluation and validation in real classrooms. We present the results of a practical evaluation of LLM-based graders for written assignments in the 2024/25 iteration of the Introduction to Bioinformatics course at the University of Ljubljana. Over the course of the semester, more than 100 students answered 36 text-based questions, most of which were automatically graded using LLMs. In a blind study, students received feedback from both LLMs and human teaching assistants without knowing the source, and later rated the quality of the feedback. We conducted a systematic evaluation of six commercial and open-source LLMs and compared their grading performance with human teaching assistants. Our results show that with well-designed prompts, LLMs can achieve grading accuracy and feedback quality comparable to human graders. Our results also suggest that open-source LLMs perform as well as commercial LLMs, allowing schools to implement their own grading systems while maintaining privacy.
Paper Structure (15 sections, 1 equation, 6 figures)

This paper contains 15 sections, 1 equation, 6 figures.

Table of Contents

  1. Introduction
  2. Study Design
  3. Prompts
  4. Preparing Grading Rubrics
  5. Results
  6. LLM Grading Accuracy
  7. Impact of Including Grading Rubric and Grading Examples
  8. Student Preferences for LLM-based Feedback
  9. Student Attitudes Towards LLM-based Grading
  10. Recommendations and Guidelines
  11. Conclusion
  12. Competing interests
  13. Author contributions statement
  14. Acknowledgments
  15. Funding

Figures (6)

  • Figure 1: Schema of student submissions graded by LLMs, based on TA-graded examples and grading rubric composed of criteria.
  • Figure 2: Prompt structure with a grading rubric and TA-graded examples. The system prompt remains unchanged between exercises, while the user prompt contains dynamic elements for each exercise, such as questions, correct answers, grading rubric, and examples. Except for student submission, all other entries are provided by the TA in advance. The response is a JSON structured text with predefined fields.
  • Figure 3: LLM performance on predicting grading criteria. TA grades represent the gold standard. 95% credible intervals (CI) of summary statistics are obtained using bootstrap samples. a) Classification accuracy of LLMs predicting each satisfied criteria as a binary classification. b) The average grading difference in prediction indicating more lenient (positive) or stringent (negative) grading by the LLM compared to TAs. c) The standard deviation of grading difference indicates consistency among models.
  • Figure 4: Importance of grading rubric and graded-examples on LLM performance. The scale relates to systematic bias with respect to TA grades. 95% credible intervals (CI) of summary statistics are obtained using bootstrap samples. a) LLM performance using only the TA-defined grading rubric in the user prompt. b) LLM performance using only TA-graded examples without the grading rubric. c) LLM performance using both a grading rubric and graded examples in the user prompt. Both rubric and examples are used in production.
  • Figure 5: Group factor in relation to student preference. Due to the correlation between grading group sample values, the TA grading group is used as a reference, and the values show an increase and a decrease in satisfaction with TA written feedback.
  • ...and 1 more figures