Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Trustworthy Difficulty Assessments: Large Language Models as Judges in Programming and Synthetic Tasks

H. M. Shadman Tabib, Jaber Ahmed Deedar

TL;DR

The paper interrogates whether a state-of-the-art LLM can reliably judge the difficulty of competitive-programming problems. It compares GPT-4o (text-only) against a LightGBM ensemble that uses both textual features and explicit numeric constraints, on 1,825 LeetCode problems, with SHAP analyses and a synthetic hard-problem protocol. Results show GPT-4o significantly underestimates real Hard problems, while LightGBM achieves strong, interpretable performance driven by numeric constraints like input size and acceptance rate; synthetic hard problems reveal inconsistencies in the LLM's internal difficulty boundary. The work highlights substantial risks of deploying LLM-based judges in education, coding platforms, or RL pipelines and advocates a hybrid, constraint-aware evaluation approach that couples LLM-derived representations with transparent numeric models.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in natural language and code generation, and are increasingly deployed as automatic judges of model outputs and learning activities. Yet, their behavior on structured tasks such as predicting the difficulty of competitive programming problems remains under-explored. We conduct a systematic comparison of GPT-4o, used purely as a natural-language difficulty assessor, against an interpretable Light-GBM ensemble trained on explicit numeric and textual features. On a dataset of 1,825 LeetCode problems labeled Easy, Medium, or Hard, LightGBM attains 86% accuracy, whereas GPT-4o reaches only 37.75%. Detailed analyses, including confusion matrices and SHAP-based interpretability, show that numeric constraints -- such as input size limits and acceptance rates -- play a crucial role in separating Hard problems from easier ones. By contrast, GPT-4o often overlooks these cues and exhibits a strong bias toward simpler categories. We further probe GPT-4o through a synthetic Hard-problem generation protocol. Surprisingly, GPT-4o labels almost all of its own synthetic Hard problems as Medium, contradicting its tendency to downgrade real Hard problems to Easy. Our findings connect to recent work on LLMs-as-judges and automatic difficulty estimation in programming and education, and highlight concrete failure modes that must be addressed before LLM-based judges can be considered trustworthy in competitive programming, educational platforms, or reinforcement-learning pipelines.

Toward Trustworthy Difficulty Assessments: Large Language Models as Judges in Programming and Synthetic Tasks

TL;DR

The paper interrogates whether a state-of-the-art LLM can reliably judge the difficulty of competitive-programming problems. It compares GPT-4o (text-only) against a LightGBM ensemble that uses both textual features and explicit numeric constraints, on 1,825 LeetCode problems, with SHAP analyses and a synthetic hard-problem protocol. Results show GPT-4o significantly underestimates real Hard problems, while LightGBM achieves strong, interpretable performance driven by numeric constraints like input size and acceptance rate; synthetic hard problems reveal inconsistencies in the LLM's internal difficulty boundary. The work highlights substantial risks of deploying LLM-based judges in education, coding platforms, or RL pipelines and advocates a hybrid, constraint-aware evaluation approach that couples LLM-derived representations with transparent numeric models.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in natural language and code generation, and are increasingly deployed as automatic judges of model outputs and learning activities. Yet, their behavior on structured tasks such as predicting the difficulty of competitive programming problems remains under-explored. We conduct a systematic comparison of GPT-4o, used purely as a natural-language difficulty assessor, against an interpretable Light-GBM ensemble trained on explicit numeric and textual features. On a dataset of 1,825 LeetCode problems labeled Easy, Medium, or Hard, LightGBM attains 86% accuracy, whereas GPT-4o reaches only 37.75%. Detailed analyses, including confusion matrices and SHAP-based interpretability, show that numeric constraints -- such as input size limits and acceptance rates -- play a crucial role in separating Hard problems from easier ones. By contrast, GPT-4o often overlooks these cues and exhibits a strong bias toward simpler categories. We further probe GPT-4o through a synthetic Hard-problem generation protocol. Surprisingly, GPT-4o labels almost all of its own synthetic Hard problems as Medium, contradicting its tendency to downgrade real Hard problems to Easy. Our findings connect to recent work on LLMs-as-judges and automatic difficulty estimation in programming and education, and highlight concrete failure modes that must be addressed before LLM-based judges can be considered trustworthy in competitive programming, educational platforms, or reinforcement-learning pipelines.

Paper Structure

This paper contains 23 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs as Judges and Automatic Evaluators
  4. LLMs, Code Generation, and Competitive Programming
  5. Automatic Difficulty Estimation in Programming and Education
  6. Interpretable Machine Learning for Trustworthy Judges
  7. Methodology
  8. Dataset and Labeling
  9. LLM Prompting and Labeling Protocol
  10. LightGBM Ensemble and Interpretability
  11. Synthetic Hard Problem Generation
  12. Results
  13. Overall Performance
  14. LLM Labeling Distributions
  15. Confusion Matrix Analysis for LightGBM
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Confusion matrix for the LightGBM ensemble on the original dataset, illustrating strong discrimination among Easy, Medium, and Hard problems.
  • Figure 2: Global SHAP bar plot indicating that constraint-related features (e.g., maximum input size, acceptance rate) dominate LightGBM's classification of Hard tasks.