Analyzing and Mitigating Surface Bias in Code Evaluation Metrics
Simantika Bhattacharjee Dristi, Matthew B. Dwyer
TL;DR
This work addresses the persistent surface bias in reference-based code evaluation metrics (CEMs) by demonstrating that popular metrics largely track superficial surface features rather than actual program semantics. It introduces LoCaL, a large-scale Python benchmark built with a differential fuzzing pipeline to quantify functional similarity without predefined test suites and to probe corner regions where CEMs fail. Across four CEMs, LoCaL reveals strong degradations in performance, especially in regions where surface similarity and functional behavior diverge, and shows LoCaL’s data can significantly improve metric robustness when used for training. The study provides a framework and dataset that can guide the development of more semantics-aware CEMs and underscores the potential of LoCaL-like data to counter surface bias in code evaluation.
Abstract
With the increasing popularity of large language models (LLMs) and LLM-based agents, reliable and effective code evaluation metrics (CEMs) have become crucial for progress across several software engineering tasks. While popular benchmarks often provide test cases to assess the correctness of generated code, crafting and executing test cases is expensive. Reference-based CEMs provide a cheaper alternative by scoring a candidate program based on its functional similarity to a reference. Although prior research has focused on reporting the weak correlation between these CEMs and functional correctness, the causes are only assumed, and plausible solutions remain unexplored. In this work, we critically evaluate four state-of-the-art reference-based CEMs, revealing their strong bias towards surface-level features rather than code functionality. Despite this surface bias, current evaluation datasets for these CEMs rarely include code pairs that are surface-similar yet functionally dissimilar, or functionally similar yet surface-dissimilar. To mitigate this gap, we propose LoCaL (Looks Can Lie), a CEM evaluation benchmark, with 3117 code pairs at both the method and program levels. Each pair is labeled with a functional similarity score and aims to target regions where CEMs are likely to perform poorly. The functional similarity scores are calculated through differential fuzzing, which eliminates the need for predefined test cases and, at the same time, improves the reliability of the scores by executing an order of magnitude more tests than prior work. We find that all four CEMs show significant performance degradation on LoCaL, compared to the baselines. Finally, based on our findings, we draw the implication that exposing CEMs to LoCaL-like data might facilitate the development of metrics that are robust to surface bias.