Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comprehensive Evaluation of Large Language Models on Software Engineering Tasks: A Multi-Task Benchmark

Go Frendi Gunawan, Mukhlis Amien

TL;DR

The paper addresses the lack of a holistic, reproducible benchmark for software engineering tasks by evaluating 11 state-of-the-art LLMs across five representative SE tasks using an automated verification framework. It introduces novel efficiency metrics, including a Tool Efficiency Ratio and a cost-estimation framework, and identifies two distinct inefficiency patterns (loop and inference) despite identical task scores. The results reveal that several models achieve perfect scores, yet exhibit dramatic variance in time, tool usage, and cost, with coding tasks saturating at 100% success and research tasks remaining more challenging. The work provides practical guidance for model selection based on task type, speed, and budget, and releases all data and tooling to enable reproducibility and future benchmarking.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in software engineering, yet comprehensive benchmarks covering diverse SE activities remain limited. We present a multi-task evaluation of 11 state-of-the-art LLMs across five representative software engineering tasks: bug fixing, feature development, code refactoring, technical copywriting, and research synthesis. Our automated verification framework measures both output quality and completion efficiency. Key findings reveal that (1) models achieving identical perfect scores exhibit 22x variation in completion time, 49x variation in tool efficiency, and 53x variation in estimated cost; (2) tool usage frequency shows no correlation with success (r = 0.077, p = 0.575) - one model used 917 tool calls while another solved the same task with 3 calls; (3) we identify two distinct inefficiency patterns: loop inefficiency and inference inefficiency; and (4) coding tasks achieve 100 percent success while research tasks present greater challenges (90.9 percent). We release all experimental data, verification scripts, and analysis code for full reproducibility.

Comprehensive Evaluation of Large Language Models on Software Engineering Tasks: A Multi-Task Benchmark

TL;DR

The paper addresses the lack of a holistic, reproducible benchmark for software engineering tasks by evaluating 11 state-of-the-art LLMs across five representative SE tasks using an automated verification framework. It introduces novel efficiency metrics, including a Tool Efficiency Ratio and a cost-estimation framework, and identifies two distinct inefficiency patterns (loop and inference) despite identical task scores. The results reveal that several models achieve perfect scores, yet exhibit dramatic variance in time, tool usage, and cost, with coding tasks saturating at 100% success and research tasks remaining more challenging. The work provides practical guidance for model selection based on task type, speed, and budget, and releases all data and tooling to enable reproducibility and future benchmarking.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in software engineering, yet comprehensive benchmarks covering diverse SE activities remain limited. We present a multi-task evaluation of 11 state-of-the-art LLMs across five representative software engineering tasks: bug fixing, feature development, code refactoring, technical copywriting, and research synthesis. Our automated verification framework measures both output quality and completion efficiency. Key findings reveal that (1) models achieving identical perfect scores exhibit 22x variation in completion time, 49x variation in tool efficiency, and 53x variation in estimated cost; (2) tool usage frequency shows no correlation with success (r = 0.077, p = 0.575) - one model used 917 tool calls while another solved the same task with 3 calls; (3) we identify two distinct inefficiency patterns: loop inefficiency and inference inefficiency; and (4) coding tasks achieve 100 percent success while research tasks present greater challenges (90.9 percent). We release all experimental data, verification scripts, and analysis code for full reproducibility.
Paper Structure (59 sections, 7 figures, 4 tables)

This paper contains 59 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Research Questions
  4. Contributions
  5. Key Findings
  6. Paper Organization
  7. Related Work
  8. Code Generation Benchmarks
  9. Software Engineering Benchmarks
  10. Multi-Task Evaluation
  11. Tool-Augmented Evaluation
  12. Efficiency and Cost Analysis
  13. Research Gap
  14. Methodology
  15. Task Categories
  16. ...and 44 more sections

Figures (7)

  • Figure 1: Average completion time by model with standard deviation. OpenAI models (blue) consistently fastest; Open/Ollama models (red) slowest.
  • Figure 2: Challenge difficulty analysis showing success rates and average duration by task type. Research tasks were most challenging.
  • Figure 3: Performance heatmap showing scores and completion times across models and challenges. Darker shading indicates longer duration.
  • Figure 4: Average tool calls by model (log scale). The 917-call anomaly for Gemini-3 Flash on bug-fix is visible as the maximum outlier.
  • Figure 5: Efficiency frontier showing speed-quality tradeoff. Circle size indicates tool call count. Models in upper-left quadrant (fast and high-scoring) represent optimal efficiency.
  • ...and 2 more figures