Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty

Leonidas Zotos, Ivo Pascal de Jong, Matias Valdenegro-Toro, Andreea Ioana Sburlea, Malvina Nissim, Hedderik van Rijn

TL;DR

This work tackles the challenge of estimating exam item difficulty by comparing state-of-the-art NLP/LLM methods against expert professors using the ground-truth metric $p^+\text{-value}$ for True/False questions across two AI courses. It introduces two NLP pipelines—direct prompt-based estimation and LLM-uncertainty–driven supervised learning—and shows that Gemini 2.5 often outperforms professors, while uncertainty-based learning can achieve the strongest performance with as few as 42 training samples. The findings demonstrate the potential of uncertainty-aware NLP approaches to augment exam design, delivering higher ranking accuracy and lower RMSE than human experts in these settings. The work also discusses limitations and the need for human-in-the-loop deployment to ensure robust and fair application across domains.

Abstract

Estimating the difficulty of exam questions is essential for developing good exams, but professors are not always good at this task. We compare various Large Language Model-based methods with three professors in their ability to estimate what percentage of students will give correct answers on True/False exam questions in the areas of Neural Networks and Machine Learning. Our results show that the professors have limited ability to distinguish between easy and difficult questions and that they are outperformed by directly asking Gemini 2.5 to solve this task. Yet, we obtained even better results using uncertainties of the LLMs solving the questions in a supervised learning setting, using only 42 training samples. We conclude that supervised learning using LLM uncertainty can help professors better estimate the difficulty of exam questions, improving the quality of assessment.

NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty

TL;DR

This work tackles the challenge of estimating exam item difficulty by comparing state-of-the-art NLP/LLM methods against expert professors using the ground-truth metric for True/False questions across two AI courses. It introduces two NLP pipelines—direct prompt-based estimation and LLM-uncertainty–driven supervised learning—and shows that Gemini 2.5 often outperforms professors, while uncertainty-based learning can achieve the strongest performance with as few as 42 training samples. The findings demonstrate the potential of uncertainty-aware NLP approaches to augment exam design, delivering higher ranking accuracy and lower RMSE than human experts in these settings. The work also discusses limitations and the need for human-in-the-loop deployment to ensure robust and fair application across domains.

Abstract

Estimating the difficulty of exam questions is essential for developing good exams, but professors are not always good at this task. We compare various Large Language Model-based methods with three professors in their ability to estimate what percentage of students will give correct answers on True/False exam questions in the areas of Neural Networks and Machine Learning. Our results show that the professors have limited ability to distinguish between easy and difficult questions and that they are outperformed by directly asking Gemini 2.5 to solve this task. Yet, we obtained even better results using uncertainties of the LLMs solving the questions in a supervised learning setting, using only 42 training samples. We conclude that supervised learning using LLM uncertainty can help professors better estimate the difficulty of exam questions, improving the quality of assessment.

Paper Structure

This paper contains 17 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Exam data
  5. Professors' estimations
  6. NLP Approaches
  7. Using Direct Estimation
  8. Using LLM Uncertainty
  9. Results
  10. Professor Performance
  11. Direct Prompting Performance
  12. Supervised Learning Performance
  13. Inter-Annotator Agreement
  14. Per Question Analysis
  15. Discussion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Inter-Annotator Agreement, calculated using Spearman's $\rho$ for the two datasets.
  • Figure 2: Estimated $p^+\text{-value}$ per question item for the best-performing professor, Gemini 2.5 and trained support vector machine using LLM Uncertainties. The Train/Test split is only relevant for the Supervised Learning Model, for which we only report its performance on the unseen question items.