Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models As MOOCs Graders

Shahriar Golchin, Nikhil Garuda, Christopher Impey, Matthew Wenger

TL;DR

This study explores the feasibility of leveraging large language models (LLMs) to replace peer grading in MOOCs and reveals a promising direction for automating grading systems for MOOCs, especially in subjects with well-defined rubrics.

Abstract

Massive open online courses (MOOCs) unlock the doors to free education for anyone around the globe with access to a computer and the internet. Despite this democratization of learning, the massive enrollment in these courses means it is almost impossible for one instructor to assess every student's writing assignment. As a result, peer grading, often guided by a straightforward rubric, is the method of choice. While convenient, peer grading often falls short in terms of reliability and validity. In this study, using 18 distinct settings, we explore the feasibility of leveraging large language models (LLMs) to replace peer grading in MOOCs. Specifically, we focus on two state-of-the-art LLMs: GPT-4 and GPT-3.5, across three distinct courses: Introductory Astronomy, Astrobiology, and the History and Philosophy of Astronomy. To instruct LLMs, we use three different prompts based on a variant of the zero-shot chain-of-thought (Zero-shot-CoT) prompting technique: Zero-shot-CoT combined with instructor-provided correct answers; Zero-shot-CoT in conjunction with both instructor-formulated answers and rubrics; and Zero-shot-CoT with instructor-offered correct answers and LLM-generated rubrics. Our results show that Zero-shot-CoT, when integrated with instructor-provided answers and rubrics, produces grades that are more aligned with those assigned by instructors compared to peer grading. However, the History and Philosophy of Astronomy course proves to be more challenging in terms of grading as opposed to other courses. Finally, our study reveals a promising direction for automating grading systems for MOOCs, especially in subjects with well-defined rubrics.

Large Language Models As MOOCs Graders

TL;DR

This study explores the feasibility of leveraging large language models (LLMs) to replace peer grading in MOOCs and reveals a promising direction for automating grading systems for MOOCs, especially in subjects with well-defined rubrics.

Abstract

Massive open online courses (MOOCs) unlock the doors to free education for anyone around the globe with access to a computer and the internet. Despite this democratization of learning, the massive enrollment in these courses means it is almost impossible for one instructor to assess every student's writing assignment. As a result, peer grading, often guided by a straightforward rubric, is the method of choice. While convenient, peer grading often falls short in terms of reliability and validity. In this study, using 18 distinct settings, we explore the feasibility of leveraging large language models (LLMs) to replace peer grading in MOOCs. Specifically, we focus on two state-of-the-art LLMs: GPT-4 and GPT-3.5, across three distinct courses: Introductory Astronomy, Astrobiology, and the History and Philosophy of Astronomy. To instruct LLMs, we use three different prompts based on a variant of the zero-shot chain-of-thought (Zero-shot-CoT) prompting technique: Zero-shot-CoT combined with instructor-provided correct answers; Zero-shot-CoT in conjunction with both instructor-formulated answers and rubrics; and Zero-shot-CoT with instructor-offered correct answers and LLM-generated rubrics. Our results show that Zero-shot-CoT, when integrated with instructor-provided answers and rubrics, produces grades that are more aligned with those assigned by instructors compared to peer grading. However, the History and Philosophy of Astronomy course proves to be more challenging in terms of grading as opposed to other courses. Finally, our study reveals a promising direction for automating grading systems for MOOCs, especially in subjects with well-defined rubrics.
Paper Structure (9 sections, 3 figures, 2 tables)

This paper contains 9 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Prompts
  5. Bootstrap Resampling
  6. Baseline
  7. Experimental Setup
  8. Results and Discussion
  9. Conclusion

Figures (3)

  • Figure 1: An illustration of the Zero-shot-CoT prompt along with answers provided by the course instructor to grade students' assignments. Each question is assessed individually for every student. We repeat this process for all questions and students, incorporating their answers into the prompt, and tasking the LLM to grade their assignments.
  • Figure 2: A representation of the Zero-shot-CoT prompt that incorporates both instructor-provided correct answers and grading rubrics for grading students' assignments. The grading process utilized in the Zero-shot-CoT with correct answers only (Figure \ref{['figure:zero-shot-cot-with-answers']}) is also applied here.
  • Figure 3: A depiction of the prompt employed to generate a rubric using GPT-4 for the Astrobiology course. We repeat this process for all the courses under study by replacing the course name, correct answer, total grade, and the question. The generated rubric is then embedded into the prompt containing Zero-shot-CoT and the correct answer for grading students' writing assignments. Specifically, the prompt displayed in Figure \ref{['figure:zero-shot-cot-with-answers-and-rubrics']} is used for grading, where the instructor-supplied rubric is substituted with an LLM-produced rubric.