CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation

TL;DR

CoCo-Bench addresses the need for a holistic, objective benchmark for code-oriented LLMs by defining four interconnected tasks—code understanding, code generation, code modification, and code review—and a difficulty-aware scoring framework. It formalizes task definitions with precise mathematical notation, introduces a weighted evaluation metric (CoCo-Score) built on a difficulty-aware pass rate, and provides extensive empirical results showing how model size and instruction-tuning affect performance across languages and tasks. The findings reveal strong task interdependencies, context-length effects, and decoding-strategy influences, offering nuanced guidance for developing robust code-oriented LLMs. Overall, CoCo-Bench offers a rigorous, real-world-aligned benchmark that differentiates model capabilities beyond existing single-task benchmarks and informs future research and practical software engineering applications.

Abstract

Large language models (LLMs) play a crucial role in software engineering, excelling in tasks like code generation and maintenance. However, existing benchmarks are often narrow in scope, focusing on a specific task and lack a comprehensive evaluation framework that reflects real-world applications. To address these gaps, we introduce CoCo-Bench (Comprehensive Code Benchmark), designed to evaluate LLMs across four critical dimensions: code understanding, code generation, code modification, and code review. These dimensions capture essential developer needs, ensuring a more systematic and representative evaluation. CoCo-Bench includes multiple programming languages and varying task difficulties, with rigorous manual review to ensure data quality and accuracy. Empirical results show that CoCo-Bench aligns with existing benchmarks while uncovering significant variations in model performance, effectively highlighting strengths and weaknesses. By offering a holistic and objective evaluation, CoCo-Bench provides valuable insights to guide future research and technological advancements in code-oriented LLMs, establishing a reliable benchmark for the field.

Figures (12)

• Figure 1: Overview of the core evaluation dimensions in CoCo-Bench. The framework assesses four critical capabilities of code LLMs: code understanding (CU), code generation (CG), code modification (CM), and code review (CR). The evaluation flow highlights the interconnected nature of these capabilities in real-world software development scenarios.
• Figure 2: Illustration of the four primary tasks in CoCo-Bench—Code Understanding (CU), Code Generation (CG), Code Modification (CM), and Code Review (CR)—each defined to evaluate the capabilities of large language models (LLMs) in software engineering.
• Figure 3: The correlation between each two of the five tasks (CUF, CUR, CG, CM, CR) on CoCo-Bench.
• Figure 4: Comparative analysis of model performance on CoCo-Bench and HumanEval benchmarks: This figure illustrates the relationship between model performance on the CoCo-Bench (CG pass@1) and HumanEval (HumanEval@1) benchmarks. The dashed trend line and shaded area indicate the general correlation between performance on the two benchmarks.
• Figure 5: Performance differences between instruct and base versions of the same models on CoCo-Bench