Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI

Tidor-Vlad Pricope

TL;DR

HardML provides a modern, rigorously curated benchmark to assess AI knowledge and reasoning in data science and machine learning through 100 challenging multi-answer multiple-choice questions. It emphasizes originality to minimize data contamination and contrasts with existing benchmarks by its up-to-date scope and difficulty, reporting substantial gaps in current models (top ~70% accuracy) and offering EasyML as a broader-access supplement. The work positions HardML alongside benchmarks like MMLU and FrontierMath as a demanding testbed for advanced reasoning, while acknowledging limitations of MCQ formats and the need for verifiable outputs. Overall, HardML advances DS/ML AI evaluation by delivering a contemporary, human-analytic benchmark with practical accessibility via neuraprep, encouraging ongoing progress tracking in rapidly evolving AI systems.

Abstract

We present HardML, a benchmark designed to evaluate the knowledge and reasoning abilities in the fields of data science and machine learning. HardML comprises a diverse set of 100 challenging multiple-choice questions, handcrafted over a period of 6 months, covering the most popular and modern branches of data science and machine learning. These questions are challenging even for a typical Senior Machine Learning Engineer to answer correctly. To minimize the risk of data contamination, HardML uses mostly original content devised by the author. Current state of the art AI models achieve a 30% error rate on this benchmark, which is about 3 times larger than the one achieved on the equivalent, well known MMLU ML. While HardML is limited in scope and not aiming to push the frontier, primarily due to its multiple choice nature, it serves as a rigorous and modern testbed to quantify and track the progress of top AI. While plenty benchmarks and experimentation in LLM evaluation exist in other STEM fields like mathematics, physics and chemistry, the subfields of data science and machine learning remain fairly underexplored.

HardML: A Benchmark For Evaluating Data Science And Machine Learning knowledge and reasoning in AI

TL;DR

HardML provides a modern, rigorously curated benchmark to assess AI knowledge and reasoning in data science and machine learning through 100 challenging multi-answer multiple-choice questions. It emphasizes originality to minimize data contamination and contrasts with existing benchmarks by its up-to-date scope and difficulty, reporting substantial gaps in current models (top ~70% accuracy) and offering EasyML as a broader-access supplement. The work positions HardML alongside benchmarks like MMLU and FrontierMath as a demanding testbed for advanced reasoning, while acknowledging limitations of MCQ formats and the need for verifiable outputs. Overall, HardML advances DS/ML AI evaluation by delivering a contemporary, human-analytic benchmark with practical accessibility via neuraprep, encouraging ongoing progress tracking in rapidly evolving AI systems.

Abstract

We present HardML, a benchmark designed to evaluate the knowledge and reasoning abilities in the fields of data science and machine learning. HardML comprises a diverse set of 100 challenging multiple-choice questions, handcrafted over a period of 6 months, covering the most popular and modern branches of data science and machine learning. These questions are challenging even for a typical Senior Machine Learning Engineer to answer correctly. To minimize the risk of data contamination, HardML uses mostly original content devised by the author. Current state of the art AI models achieve a 30% error rate on this benchmark, which is about 3 times larger than the one achieved on the equivalent, well known MMLU ML. While HardML is limited in scope and not aiming to push the frontier, primarily due to its multiple choice nature, it serves as a rigorous and modern testbed to quantify and track the progress of top AI. While plenty benchmarks and experimentation in LLM evaluation exist in other STEM fields like mathematics, physics and chemistry, the subfields of data science and machine learning remain fairly underexplored.
Paper Structure (21 sections, 4 figures, 2 tables)

This paper contains 21 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Data collection
  3. The collection pipeline
  4. Question difficulty
  5. Dataset composition
  6. Dataset Statistics
  7. Comparison to related benchmarks
  8. Sample questions from HardML
  9. Results
  10. Accuracy on HardML
  11. Human performance on HardML
  12. Accuracy on MMLU and EasyML
  13. Related work
  14. Multitask Language Understanding Benchmarks
  15. Benchmarks for Advanced Reasoning
  16. ...and 6 more sections

Figures (4)

  • Figure 1: Comparison of error rate across 3 DS&ML benchmarks. While existing benchmarks are approaching saturation, HardML keeps an average level above saturation, in line with benchmarks from other fields like MathVista 23 or AIME 24 despite the multiple-choice nature
  • Figure 2: Solved questions in HardML
  • Figure 3: Solved questions in EasyML
  • Figure 4: Solved questions in MMLU [ML]