Uncovering Competency Gaps in Large Language Models and Their Benchmarks

TL;DR

This work tackles the problem that aggregated benchmark scores obscure fine-grained competency gaps in large language models and mask coverage imbalances across benchmarks. It proposes Competency Gaps (CG), a representation-grounded evaluation method that uses pre-trained sparse autoencoders to map internal model representations to a fixed concept dictionary, enabling concept-level decomposition of benchmark scores. CG defines activation-based metrics for benchmark gaps and model gaps, demonstrates its application on two open-source models across ten benchmarks, and reveals sycophantic and safety-related concept gaps as well as skewed benchmark coverage. The approach is complemented by an interactive web app and open-source code, offering a practical, scalable path to diagnose and iteratively improve both benchmarks and models for real-world use.

Abstract

The evaluation of large language models (LLMs) relies heavily on standardized benchmarks. These benchmarks provide useful aggregated metrics for a given capability, but those aggregated metrics can obscure (i) particular sub-areas where the LLMs are weak ("model gaps") and (ii) imbalanced coverage in the benchmarks themselves ("benchmark gaps"). We propose a new method that uses sparse autoencoders (SAEs) to automatically uncover both types of gaps. By extracting SAE concept activations and computing saliency-weighted performance scores across benchmark data, the method grounds evaluation in the model's internal representations and enables comparison across benchmarks. As examples demonstrating our approach, we applied the method to two popular open-source models and ten benchmarks. We found that these models consistently underperformed on concepts that stand in contrast to sycophantic behaviors (e.g., politely refusing a request or asserting boundaries) and concepts connected to safety discussions. These model gaps align with observations previously surfaced in the literature; our automated, unsupervised method was able to recover them without manual supervision. We also observed benchmark gaps: many of the evaluated benchmarks over-represented concepts related to obedience, authority, or instruction-following, while missing core concepts that should fall within their intended scope. In sum, our method offers a representation-grounded approach to evaluation, enabling concept-level decomposition of benchmark scores. Rather than replacing conventional aggregated metrics, CG complements them by providing a concept-level decomposition that can reveal why a model scored as it did and how benchmarks could evolve to better reflect their intended scope. Code is available at https://competency-gaps.github.io.

Figures (27)

• Figure 1: Competency Gaps (CG) Method Overview. CG decomposes LLM evaluation into interpretable benchmark gaps and model gaps using the concept dictionary learned by a sparse autoencoder (SAE), a subset of which is visualized above. (a) Benchmark Gaps quantify how much benchmarks activate individual concepts and hence surface underrepresented regions. (b) Model Gaps project model performance into concept space, yielding per-concept scores for individual benchmarks and for entire multi-benchmark evaluation suites.
• Figure 2: Recommended workflows for applying the Competency Gaps (CG) method in production.
• Figure 3: Cross-Benchmark Coverage. The distribution of $\bm{\mathit{X}}_{\text{bench}}^{(c)}$ scores obtained for the $10$ evaluated benchmarks, using the SAE of Llama 3.1 8B. This distribution exhibits strong left skew (most concepts have low coverage), and avg. performance is strongly dominated by a few concepts with high coverage (high $\bm{\mathit{X}}_{\text{bench}}^{(c)}$). Similar skewed distributions were observed for individual benchmarks (Appendix Figure \ref{['fig:bench_lvl_coverage']}). The orange curve shows a similar analysis conducted through the activations and SAE concepts of Gemma 2 2B (see Section \ref{['subsec:robustness']}).
• Figure 4: Cross-Benchmark Model Performance. The distribution of $\bm{\mathit{X}}_{\text{model}}^{(c)}$ scores obtained for Llama 3.1 8B across the $10$ evaluated benchmarks. The model exhibited high variance in performance across concepts. We observe particularly high performance for some concepts; these tend to include concepts related to coding, data handling, instruction following, and expressing positive sentiment toward the user. The orange curve shows a similar analysis conducted through the activations and SAE concepts of Gemma 2 2B (see Section \ref{['subsec:robustness']}).
• Figure 5: Model Gap Illustrated on Specific Benchmark Datapoints. Example LogicBench and WinoGrande items associated with “intuitive understanding” concepts (left: 64113, right: 64540). Llama 3.1 8B answered both incorrectly, consistent with these concepts being model gaps.