Vibe Checker: Aligning Code Evaluation with Human Preference

TL;DR

This work addresses the gap between functional correctness and human preferences in coding by introducing VeriCode, a taxonomy of 30 verifiable non-functional instructions with deterministic verifiers, and Vibe Checker, a testbed that augments standard benchmarks to jointly assess instruction following (IF) and functionality. The authors demonstrate that non-functional constraints can cause functional regressions and that IF, especially in multi-turn interactions, better aligns evaluations with real-world preferences. A composite metric combining $IF$ and functional correctness correlates more strongly with human preferences than either metric alone, highlighting the need to balance both aspects in benchmarking and training. The proposed VeriCode + Vibe Checker framework offers a concrete, scalable path to develop models that produce code that not only works but also adheres to style, documentation, error handling, and API constraints preferred by human programmers.

Abstract

Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check is tied to real-world human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking the non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check that represents human preference in coding besides functional correctness. To quantify models' code instruction following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with corresponding deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in Vibe Checker, a testbed to assess both code instruction following and functional correctness. Upon evaluating 31 leading LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit clear functional regression. Most importantly, a composite score of functional correctness and instruction following correlates the best with human preference, with the latter emerging as the primary differentiator on real-world programming tasks. Our work identifies core factors of the vibe check, providing a concrete path for benchmarking and developing models that better align with user preferences in coding.

Figures (15)

• Figure 1: Vibe check goes beyond functionality, requiring code to satisfy non-functional instructions such as coding style and logic patterns, which are also key factors of human preference.
• Figure 2: Our evaluation protocol simulates two real-world interaction patterns: single-turn generation, where all instructions are given upfront in one prompt, and multi-turn editing, where instructions are introduced sequentially to refine a solution. Both are measured for functionality and IF.
• Figure 3: Trends averaged over all evaluated models. As the number of instructions increases, functional regression grows steadily, while the task-level IF score drops markedly. Single-turn generation better preserves functionality, whereas multi-turn editing achieves higher instruction following.
• Figure 4: Average instruction-level IF trends by instruction position.
• Figure 5: Human preference aligns best with a mix of IF and functionality. We correlate LMArena coding Elo with a composite score $\alpha\,\text{IF} + (1-\alpha)\,\text{Func}$, where $\alpha\in[0,1]$ is the weight on IF (x-axis). The peak correlation (starred) for both benchmarks is achieved with a mixture of the two metrics.