SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

Abstract

Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear. We present SWE-Skills-Bench, the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering (SWE). It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. We introduce a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Our results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, and the average gain is only +1.2%. Token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance conflicting with project context. These findings suggest that agent skills are a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility. SWE-Skills-Bench provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents. SWE-Skills-Bench is available at https://github.com/GeniusHTX/SWE-Skills-Bench.

Figures (8)

• Figure 1: Illustration of how agent skills are used in a software engineering workflow. Given a natural-language requirement, the LLM-based agent selects the most relevant skill from its skill library, including skills such as writing code, running tests, debugging, creating pull requests, and deploying, and injects it into the context window. The agent then executes a series of SWE actions to produce the final software artifacts (such as code) that fulfill the requirement.
• Figure 2: The distribution of the curated skills and generated tasks.
• Figure 3: Overview of the SWE-Skills-Bench construction pipeline. We begin with 84,192 public skills and narrow them down through three filtering stages: category selection, semantic filtering, and feasibility screening. This process yields 49 SWE skills (Stage 1). Next, for each skill, we identify a matching GitHub project and generate 565 task instances of the form $(R, E, P, S)$ (Stage 2). For each criterion in the requirements document $P$, we build deterministic verifiers using pytest unit tests (Stage 3). Finally, we run a paired evaluation that compares agent performance with and without the SKILL.md file, allowing us to measure the effectiveness of the skill (Stage 4).
• Figure 4: The pipeline of task instance generation.
• Figure 5: Context interference in the linkerd-patterns skill ($\Delta P = -9.1\%$). The task requires a Server CRD and a ServerAuthorization CRD enforcing mTLS identity verification for a gRPC service. Left: Template 5 from the injected skill, which near-matches the task but encodes different concrete values: API version v1beta1 with proxyProtocol: HTTP/1, and multiple authorization modes (meshTLS, unauthenticated, and CIDR-based). Center: Without the skill, the agent reasons from first principles and produces a correct solution using v1beta3, gRPC, and standard meshTLS.serviceAccounts. Right: With the skill, the agent's output degrades through three stages, each traceable to a specific region of the template (matched by circled numbers): Surface anchoring, the agent copies v1beta1 and HTTP/1 verbatim; Hallucination, while reconciling the template's mixed authorization modes, the agent fabricates a nonexistent rules/metricsServers field; Concept bleed, the template's NetworkPolicy example causes the agent to append an unrequested resource, conflating Linkerd-level and Kubernetes-level authorization.