Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kong, Xin Lan, Jiachen Li, Songlin Li, Yijiang Li, Yueqian Lin, Xinyi Liu, Xuanqing Liu, Haoran Lyu, Ze Ma, Bowei Wang, Runhui Wang, Tianyu Wang, Wengao Ye, Yue Zhang, Hanwen Xing, Yiqi Xue, Steven Dillmann, Han-chung Lee

TL;DR

SkillsBench introduces the first comprehensive framework for evaluating Agent Skills as first-class artifacts, systematically measuring their impact across diverse tasks and domains. By comparing no-Skills, curated-Skills, and self-generated-Skills conditions on a large, deterministic benchmark with multiple harnesses and models, the study finds that curated Skills yield substantial but heterogeneous gains (average +16.2 percentage points) while self-generated Skills offer little to no benefit. The results show that focused Skills (2–3 modules) outperform exhaustive documentation, and that smaller models paired with Skills can rival or surpass larger models without Skills, underscoring the context-dependent nature of Skill efficacy. These findings provide concrete guidance for Skill authoring, evaluation, and deployment, and establish open infrastructure for principled Skill design across agent systems.

Abstract

Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills. We test 7 agent-model configurations over 7,308 trajectories. Curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain (+4.5pp for Software Engineering to +51.9pp for Healthcare) and 16 of 84 tasks show negative deltas. Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming. Focused Skills with 2--3 modules outperform comprehensive documentation, and smaller models with Skills can match larger models without them.

SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

TL;DR

SkillsBench introduces the first comprehensive framework for evaluating Agent Skills as first-class artifacts, systematically measuring their impact across diverse tasks and domains. By comparing no-Skills, curated-Skills, and self-generated-Skills conditions on a large, deterministic benchmark with multiple harnesses and models, the study finds that curated Skills yield substantial but heterogeneous gains (average +16.2 percentage points) while self-generated Skills offer little to no benefit. The results show that focused Skills (2–3 modules) outperform exhaustive documentation, and that smaller models paired with Skills can rival or surpass larger models without Skills, underscoring the context-dependent nature of Skill efficacy. These findings provide concrete guidance for Skill authoring, evaluation, and deployment, and establish open infrastructure for principled Skill design across agent systems.

Abstract

Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills. We test 7 agent-model configurations over 7,308 trajectories. Curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain (+4.5pp for Software Engineering to +51.9pp for Healthcare) and 16 of 84 tasks show negative deltas. Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming. Focused Skills with 2--3 modules outperform comprehensive documentation, and smaller models with Skills can match larger models without them.
Paper Structure (121 sections, 2 equations, 13 figures, 18 tables)

This paper contains 121 sections, 2 equations, 13 figures, 18 tables.

Table of Contents

  1. Introduction
  2. SkillsBench
  3. Skills Specification
  4. Task Specification
  5. Dataset Construction
  6. Contributing Principles
  7. Human-Authored Instructions.
  8. Skill Generality.
  9. Deterministic Verification.
  10. Automated Validation.
  11. Human Review.
  12. Leakage Prevention.
  13. Benchmark Composition
  14. Experimental Setup
  15. Agent Harnesses
  16. ...and 106 more sections

Figures (13)

  • Figure 1: Agent architecture stack and resolution rates across 7 agent-model configurations on 84 tasks. Curated Skills (beige) improve performance by +16.2pp on average; self-generated Skills (amber) provide negligible or negative benefit.
  • Figure 2: SkillsBench pipeline overview.Phase 1 (Benchmark Construction): We aggregate Skills from three sources---open-source repositories (12,847), the Claude Code ecosystem (28,412), and corporate partners (5,891)---yielding 47,150 unique Skills after deduplication. In parallel, 322 contributors submit 105 candidate tasks. Phase 2 (Quality Filtering): Each task undergoes automated checks (structural validity, AI detection, leakage audit) and human review (data validity, task realism, oracle quality, Skill quality, anti-cheating), producing 84 tasks spanning 11 domains. Phase 3 (Evaluation): Tasks are executed under three conditions (no Skills, with curated Skills, self-generated Skills) across three commercial agent harnesses (Claude Code, Gemini CLI, Codex CLI). Deterministic pytest verifiers produce pass/fail outcomes; 7 agent-model configurations yield 7,308 trajectories, with curated Skills providing +12.66pp average improvement.
  • Figure 3: SkillsBench consists of tasks spanning 11 domains.
  • Figure 4: Pareto frontier of pass rate vs. cost across model-harness configurations. Filled markers indicate with-Skills conditions; hollow markers indicate without-Skills. Skills shift the Pareto frontier upward, with Gemini 3 Flash and Claude Opus dominating the with-Skills frontier. Cost positions in this figure reflect the evaluation infrastructure's pricing model. Trajectory analysis reveals that Flash consumes 2.3$\times$ more input tokens per task than Pro (1.08M vs. 0.47M), a compensatory strategy where the smaller model substitutes iterative exploration for reasoning depth. At official API pricing ($0.50 vs. $2.00 per 1M input tokens), Flash's 4$\times$ lower per-token cost more than offsets this higher volume, making Flash 44% cheaper per task ($0.55 vs. $0.98).
  • Figure 5: Temporal dynamics of Skill creation over 136 days. Daily additions (bars, left axis) remained modest through late 2025, then surged to a peak of 18,904 in January 2026. The cumulative curve (line, right axis) reflects exponential-like growth, reaching 84,192 total Skills.
  • ...and 8 more figures