OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution

TL;DR

OmniGIRL advances GitHub issue resolution benchmarking by introducing a multilingual, multimodal, and multi-domain dataset with 959 task instances across four languages and diverse domains. It constructs realistic task instances through a five-stage pipeline involving language/repository selection, PR data collection, task construction, execution-based verification, and image filtering, complemented by ground-truth patches and optional website links. Evaluation with three state-of-the-art LLMs and three baselines reveals that current approaches achieve limited success (e.g., the top resolve rate around 8.6%), with significant drops on tasks requiring image understanding and cross-file modifications. The work analyzes root causes—parsing failures, cross-file challenges, and language-specific structural issues—and demonstrates that prompt engineering and method improvements can mitigate some failures, offering a robust dataset and analysis to guide future improvements in multilingual multimodal code repair and issue resolution.

Abstract

The GitHub issue resolution task aims to resolve issues reported in repositories automatically. With advances in large language models (LLMs), this task has gained increasing attention, and several benchmarks are proposed to evaluate the issue resolution ability of LLMs. However, existing benchmarks have three main limitations. First, current benchmarks focus on a single programming language, limiting the evaluation of issues from repositories across different languages. Second, they usually cover a narrow range of domains, which may fail to represent the diversity of real-world issues. Third, existing benchmarks rely solely on textual information in issue descriptions, overlooking multimodal information such as images in issues. In this paper, we propose OmniGIRL, a GitHub Issue ResoLution benchmark that is multilingual, multimodal, and multi-domain. OmniGIRL includes 959 task instances, which are collected from repositories across four programming languages (i.e., Python, JavaScript, TypeScript, and Java) and eight different domains. Our evaluation shows that current LLMs show limited performances on OmniGIRL. Notably, the best-performing model, GPT-4o, resolves only 8.6% of the issues. Besides, we find that current LLMs struggle to resolve issues requiring understanding images. The best performance is achieved by Claude-3.5-Sonnet, which resolves only 10.5% of the issues with image information. Finally, we analyze the reasons behind current LLMs' failure on OmniGIRL, providing insights for future improvements.

Figures (6)

• Figure 1: Overview of the evaluation pipeline of GitHub issue resolution.
• Figure 2: Overview of benchmark construction.
• Figure 3: An example of task instance tailwindlabs__tailwindcss-10212.
• Figure 4: Examples of images contained in issue descriptions.
• Figure 5: An example of website links within an issue.