Evaluating and Aligning CodeLLMs on Human Preference

TL;DR

CodeArena introduces a human-curated benchmark to evaluate alignment between code-focused LLM responses and human preferences, complemented by SynCode-Instruct, a large synthetic instruction corpus. The study demonstrates substantial performance gaps between open-source and closed-source LLMs on human-preference metrics and shows that large-scale synthetic instruction fine-tuning can significantly improve results, especially on alignment-focused benchmarks. By evaluating 40+ LLMs across 44 programming languages and using LLM-based judgments to gauge preference alignment, the work highlights practical paths toward more user-friendly code assistants and emphasizes the value of human-centric evaluation in code generation tasks.

Abstract

Code large language models (codeLLMs) have made significant strides in code generation. Most previous code-related benchmarks, which consist of various programming exercises along with the corresponding test cases, are used as a common measure to evaluate the performance and capabilities of code LLMs. However, the current code LLMs focus on synthesizing the correct code snippet, ignoring the alignment with human preferences, where the query should be sampled from the practical application scenarios and the model-generated responses should satisfy the human preference. To bridge the gap between the model-generated response and human preference, we present a rigorous human-curated benchmark CodeArena to emulate the complexity and diversity of real-world coding tasks, where 397 high-quality samples spanning 40 categories and 44 programming languages, carefully curated from user queries. Further, we propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B tokens) by scaling instructions from the website to verify the effectiveness of the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder totally trained on synthetic instruction data can achieve top-tier performance of open-source code LLMs. The results find performance differences between execution-based benchmarks and CodeArena. Our systematic experiments of CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring the importance of the human preference alignment.\footnote{\url{https://codearenaeval.github.io/ }}

Figures (10)

• Figure 1: A comparison between the GPT4o with better human preference and Qwen2.5-Coder-7B-Instruct. Qwen2.5-Coder-7B-Instruct solves the user question by simply replying with the code snippet without details.
• Figure 2: Task types of CodeArena.
• Figure 3: Statistics of programming languages in CodeArena.
• Figure 4: Number of samples of different difficulties (Easy/Medium/Hard) across categories in CodeArena.
• Figure 5: Overview of the CodeArena creation benchmark. We first collect the online code Q&A and code-related raw text from the website. We cluster the code-related data and classify them into different categories using LLM. We uniformly sample the samples from different subtasks as the seed data for manual annotation.