HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Class

TL;DR

HARDMath2 addresses the gap in evaluating LLMs on applied, approximation-based mathematics by introducing a student-generated benchmark of 211 problems across boundary layers, WKB, nonlinear PDEs, and oscillatory integrals. The dataset is created and verified through a collaboration between students and instructors in a Harvard graduate course, and evaluated using an automated parsing and numerical-ground-truth framework, ensuring objective grading. Key findings show that frontier models still struggle with many problems, while students learn from interacting with models to craft harder questions, suggesting both a methodological contribution to AI evaluation and a pedagogical boost for mathematical training. The work holds practical significance for measuring and improving mathematical reasoning in real-world scientific and engineering contexts and proposes a scalable approach to extending such benchmarks to other quantitative disciplines.

Abstract

Large language models (LLMs) have shown remarkable progress in mathematical problem-solving, but evaluation has largely focused on problems that have exact analytical solutions or involve formal proofs, often overlooking approximation-based problems ubiquitous in applied science and engineering. To fill this gap, we build on prior work and present HARDMath2, a dataset of 211 original problems covering the core topics in an introductory graduate applied math class, including boundary-layer analysis, WKB methods, asymptotic solutions of nonlinear partial differential equations, and the asymptotics of oscillatory integrals. This dataset was designed and verified by the students and instructors of a core graduate applied mathematics course at Harvard. We build the dataset through a novel collaborative environment that challenges students to write and refine difficult problems consistent with the class syllabus, peer-validate solutions, test different models, and automatically check LLM-generated solutions against their own answers and numerical ground truths. Evaluation results show that leading frontier models still struggle with many of the problems in the dataset, highlighting a gap in the mathematical reasoning skills of current LLMs. Importantly, students identified strategies to create increasingly difficult problems by interacting with the models and exploiting common failure modes. This back-and-forth with the models not only resulted in a richer and more challenging benchmark but also led to qualitative improvements in the students' understanding of the course material, which is increasingly important as we enter an age where state-of-the-art language models can solve many challenging problems across a wide domain of fields.

Figures (5)

• Figure 1: Flowchart of the problem-generation and validation process. Problem creation and validation happen on a collaborative Google Sheet, which includes custom functionality to send problems to a server for LLM querying and evaluation.
• Figure 2: Problems are collected from students in a Google Sheet, which contains fields for all relevant aspects of the problem and solution, including the prompt passed to the LLM, the regime of interest, and additional parameters. Each student submitted a Colab notebook with their problem demonstrating a numerical comparison of their analytic solution to a full numerical solution, which could then be checked by student verifiers for accuracy. Then, students could instantly run an LLM on their problem (with standardized formatting and solution parsing automatically applied).
• Figure 3: Problem type distribution and associated canonical solution forms in HARDMath2.
• Figure 4: Model performance on HARDMath2. While (a) shows overall success rates by model, (b) shows significant differences in performance across problem types.
• Figure 5: Visual comparison of numerical and approximate analytical solutions to a sample boundary value problem for solution verification.