SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

TL;DR

SWE-Lancer introduces a real-world, monetized benchmark for freelance software engineering, compiling 1,488 Upwork tasks worth $1M across IC SWE and SWE Manager roles. The evaluation uses end-to-end Playwright tests and a Docker-based harness to map model performance to actual payouts, revealing that frontier LLMs still struggle to solve most tasks, especially IC problems, while Manager-level decisions are somewhat easier. The strongest model (Claude 3.5 Sonnet) achieves notable but incomplete coverage on the Diamond split, earning about $208k of $501k possible, and over $400k on the full dataset, underscoring substantial gaps between current capabilities and real-world economic potential. By open-sourcing SWE-Lancer Diamond, the authors provide a framework to study AI's economic impact and guide future improvements in full-stack automation, agentic safety, and deployment considerations.

Abstract

We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at \$1 million USD total in real-world payouts. SWE-Lancer encompasses both independent engineering tasks--ranging from \$50 bug fixes to \$32,000 feature implementations--and managerial tasks, where models choose between technical implementation proposals. Independent tasks are graded with end-to-end tests triple-verified by experienced software engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model performance and find that frontier models are still unable to solve the majority of tasks. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond (https://github.com/openai/SWELancer-Benchmark). By mapping model performance to monetary value, we hope SWE-Lancer enables greater research into the economic impact of AI model development.

Figures (12)

• Figure 1: Evaluation flow for IC SWE tasks; the model only earns the payout if all applicable tests pass.
• Figure 2: Evaluation flow for SWE Manager tasks; during proposal selection, the model has the ability to browse the codebase.
• Figure 3: Breakdown of the different types of IC SWE issues in SWE-Lancer with examples from the Diamond Set. Blue categories on the left represent task topics, while green categories at right represent task types.
• Figure 4: SWE-Lancer issues are dynamically priced based on real-world difficulty. In the example above Expensify23Isue14958, SWE managers rejected 5 early proposals that did not appropriately address edge cases. The initial request priced at $1,000 was increased to $8,000 over four weeks until it was solved.
• Figure 5: Total payouts earned by each model on the full SWE-Lancer dataset including both IC SWE and SWE Manager tasks.