Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing GPT Performance in a Proof-Based University-Level Course Under Blind Grading

Ming Ding, Rasmus Kyng, Federico Solda, Weixuan Yuan

TL;DR

The paper tackles how well contemporary LLMs, specifically GPT-4o and o1-preview, solve complex, proof-based undergraduate problems under realistic take-home-exam conditions with blind grading. By collecting four exercises across two exams and comparing AI outputs with student work using naive prompts and minimal instructor input, the authors perform both coarse score analysis and fine-grained error inspection. They find that o1-preview often passes and occasionally matches or exceeds student performance, while GPT-4o generally falls short, with both models prone to unjustified or misleading reasoning and occasional mathematical errors. The work highlights the importance of AI-aware assessment design, robust grading rubrics, and exam formats that mitigate AI-assisted cheating while revealing concrete weaknesses in current LLM reasoning capabilities. Overall, the study provides a realistic lower-bound benchmark for AI in education and suggests careful policy and question-design adaptations to maintain educational integrity and learning outcomes.

Abstract

As large language models (LLMs) advance, their role in higher education, particularly in free-response problem-solving, requires careful examination. This study assesses the performance of GPT-4o and o1-preview under realistic educational conditions in an undergraduate algorithms course. Anonymous GPT-generated solutions to take-home exams were graded by teaching assistants unaware of their origin. Our analysis examines both coarse-grained performance (scores) and fine-grained reasoning quality (error patterns). Results show that GPT-4o consistently struggles, failing to reach the passing threshold, while o1-preview performs significantly better, surpassing the passing score and even exceeding the student median in certain exercises. However, both models exhibit issues with unjustified claims and misleading arguments. These findings highlight the need for robust assessment strategies and AI-aware grading policies in education.

Assessing GPT Performance in a Proof-Based University-Level Course Under Blind Grading

TL;DR

The paper tackles how well contemporary LLMs, specifically GPT-4o and o1-preview, solve complex, proof-based undergraduate problems under realistic take-home-exam conditions with blind grading. By collecting four exercises across two exams and comparing AI outputs with student work using naive prompts and minimal instructor input, the authors perform both coarse score analysis and fine-grained error inspection. They find that o1-preview often passes and occasionally matches or exceeds student performance, while GPT-4o generally falls short, with both models prone to unjustified or misleading reasoning and occasional mathematical errors. The work highlights the importance of AI-aware assessment design, robust grading rubrics, and exam formats that mitigate AI-assisted cheating while revealing concrete weaknesses in current LLM reasoning capabilities. Overall, the study provides a realistic lower-bound benchmark for AI in education and suggests careful policy and question-design adaptations to maintain educational integrity and learning outcomes.

Abstract

As large language models (LLMs) advance, their role in higher education, particularly in free-response problem-solving, requires careful examination. This study assesses the performance of GPT-4o and o1-preview under realistic educational conditions in an undergraduate algorithms course. Anonymous GPT-generated solutions to take-home exams were graded by teaching assistants unaware of their origin. Our analysis examines both coarse-grained performance (scores) and fine-grained reasoning quality (error patterns). Results show that GPT-4o consistently struggles, failing to reach the passing threshold, while o1-preview performs significantly better, surpassing the passing score and even exceeding the student median in certain exercises. However, both models exhibit issues with unjustified claims and misleading arguments. These findings highlight the need for robust assessment strategies and AI-aware grading policies in education.
Paper Structure (45 sections, 1 theorem, 16 equations, 5 figures, 3 tables)

This paper contains 45 sections, 1 theorem, 16 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tests against standard datasets.
  4. The use of LLMs in higher education.
  5. Tests with human comparison.
  6. High-difficulty datasets.
  7. Methodology
  8. Experiment Design
  9. Course and examination information.
  10. Data collection and grading process.
  11. AI-solution generation.
  12. Highlights of the Experiment Settings
  13. High-difficulty benchmark.
  14. Fine-grained expert solution evaluation.
  15. Realistic educational condition.
  16. ...and 30 more sections

Key Result

Theorem B.1

Let $A(y)$ be an $n\times n$ matrix where the entries $A_{i,j}$ are polynomials over $y$ with coefficients in a finite field $F$ and have degree at most $d$. Then we can compute $\det(A(y))$ in $O(dM(n)(\log (n) + \log(d))^{50})$ operations.

Figures (5)

  • Figure 1: Score per exercise.
  • Figure 2: Vertical lines are added at the extreme points of circles and at their intersections with each other. The vertices of the final subdivision are in yellow.
  • Figure 3: A scheme of a NOT-OR circuit
  • Figure 4: A $3 \times 2$ pizza and slicing of the pizza into three slices each containing one mushroom square and one pepper square.
  • Figure 5: The six options of permutations on an (even) cycle. Notice that the same permutation types exist on a longer (even) cycle. The red arrows go from vertex $i$ to vertex $\pi_1(i)$ while the blue arrows go from vertex $i$ to vertex $\pi_2(i)$.

Theorems & Definitions (1)

  • Theorem B.1