ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models

TL;DR

This work addresses a gap in LLM benchmarking by moving beyond task success to diagnose why models fail. It introduces ErrorMap, a two-stage, model-oriented error analysis pipeline that converts failures into a structured taxonomy, and ErrorAtlas, a static 17‑category taxonomy derived from 83 models across 35 datasets. The approach enables deeper interpretation, cross-benchmark comparisons, and targeted model debugging, with validations showing strong coverage, high labeling accuracy, and robust results under varied configurations. By surfacing underexplored failure modes and supporting end-to-end applicability—from development to benchmarking—ErrorMap and ErrorAtlas offer a practical framework for more reliable model improvement and evaluation.

Abstract

Large Language Models (LLM) benchmarks tell us when models fail, but not why they fail. A wrong answer on a reasoning dataset may stem from formatting issues, calculation errors, or dataset noise rather than weak reasoning. Without disentangling such causes, benchmarks remain incomplete and cannot reliably guide model improvement. We introduce ErrorMap, the first method to chart the sources of LLM failure. It extracts a model's unique "failure signature", clarifies what benchmarks measure, and broadens error identification to reduce blind spots. This helps developers debug models, aligns benchmark goals with outcomes, and supports informed model selection. ErrorMap works on any model or dataset with the same logic. Applying our method to 35 datasets and 83 models we generate ErrorAtlas, a taxonomy of model errors, revealing recurring failure patterns. ErrorAtlas highlights error types that are currently underexplored in LLM research, such as omissions of required details in the output and question misinterpretation. By shifting focus from where models succeed to why they fail, ErrorMap and ErrorAtlas enable advanced evaluation - one that exposes hidden weaknesses and directs progress. Unlike success, typically measured by task-level metrics, our approach introduces a deeper evaluation layer that can be applied globally across models and tasks, offering richer insights into model behavior and limitations. We make the taxonomy and code publicly available with plans to periodically update ErrorAtlas as new benchmarks and models emerge.

Figures (5)

• Figure 1: ErrorMap Schema: Two Method Stages. The first stage performs a one-time analysis of each incorrect prediction at the instance level. The second stage constructs a layered taxonomy through recursive iterations, based on either the maximum grouping depth or a defined layer threshold.
• Figure 2: Error distributions for models across the $10$ most prevalent ErrorAtlas error categories on the Capabilities benchmark. The total number of errors for each model appears at the center of its corresponding donuts chart. Each error count represents roughly $10\%$ of the model's total errors on Capabilities.
• Figure 3: Differences in number of errors per category between Gemini 1.5 flash and Pro on the capabilities benchmark in HELM. X-axis represents the change in error frequency, highlighting areas of improvement or regression.
• Figure 4: Differences in error category distributions among three leading models on the MMLU-Pro dataset. Asterisks (*) indicate bars compared in statistical significance test (see App. Table \ref{['tab:significance_best_worst']})
• Figure 5: Differences in error category distributions across domain categories in MMLU-Pro.