OmniCode: A Benchmark for Evaluating Software Engineering Agents

TL;DR

OmniCode tackles the fragmentation of existing coding benchmarks by introducing a multi-task, multi-language benchmark that covers bug fixing, test generation, code-review responses, and style fixing. It combines real-world GitHub data with synthetic task generation and rigorous evaluation protocols to assess end-to-end software engineering capabilities of LLM-powered agents. Across SWE-Agent and Aider, results reveal strong performance in Python-style fixing but notable gaps in test generation and cross-language robustness, with patch complexity emerging as a key predictor of success. The work provides a foundation for more comprehensive assessment and directs future development toward agents capable of holistic software development tasks.

Abstract

LLM-powered coding agents are redefining how real-world software is developed. To drive the research towards better coding agents, we require challenging benchmarks that can rigorously evaluate the ability of such agents to perform various software engineering tasks. However, popular coding benchmarks such as HumanEval and SWE-Bench focus on narrowly scoped tasks such as competition programming and patch generation. In reality, software engineers have to handle a broader set of tasks for real-world software development. To address this gap, we propose OmniCode, a novel software engineering benchmark that contains a broader and more diverse set of task categories beyond code or patch generation. Overall, OmniCode contains 1794 tasks spanning three programming languages (Python, Java, and C++) and four key categories: bug fixing, test generation, code review fixing, and style fixing. In contrast to prior software engineering benchmarks, the tasks in OmniCode are (1) manually validated to eliminate ill-defined problems, and (2) synthetically crafted or recently curated to avoid data leakage issues, presenting a new framework for synthetically generating diverse software tasks from limited real-world data. We evaluate OmniCode with popular agent frameworks such as SWE-Agent and show that while they may perform well on bug fixing for Python, they fall short on tasks such as Test Generation and in languages such as C++ and Java. For instance, SWE-Agent achieves a maximum of 20.9% with DeepSeek-V3.1 on Java Test Generation tasks. OmniCode aims to serve as a robust benchmark and spur the development of agents that can perform well across different aspects of software development. Code and data are available at https://github.com/seal-research/OmniCode.

Figures (15)

• Figure 1: OmniCode synthetically builds multiple tasks out of a base dataset to holistically evaluate software engineering agents. Four different types of tasks that we consider: Bug fixing, test generation, responding to code review, and enforcing style guidelines.
• Figure 2: In the Test Generation task, we evaluate proposed test patch against both the ground truth (gold) patch, as well as several meaningful, but incorrect, bad patches. A test is only considered correct if it passes for the gold test, but fails for all bad patches.
• Figure 3: In the task of responding to Code Review, an initial incorrect patch is provided, which contains a meaningful attempt at solving a given problem. This attempt is then reviewed by a human or an LLM, and a review report is generated. Utilizing this report, the LLM is tasked with correcting the initial approach, which is then validated with the normal testing suite.
• Figure 4: In the Style Fix task, we first create task instances by running a style check tool on the whole repository and grouping local issues into instances. The agent is then tasked with fixing these instances, with the proposed fix being evaluated by running the style checker again.
• Figure 5: Distribution of patch complexity scores for resolved versus unresolved instances.