CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation

TL;DR

This work tackles the challenge of evaluating generated code without relying on reference implementations or test cases by introducing CODE-DiTing, a reasoning-based, explainable evaluation metric. It combines data distillation from a large reasoning model to compact CODE-DiTing models (1.5B and 7B) via PiSSA-initialized LoRA fine-tuning, resulting in high accuracy and efficiency with a majority-vote inference strategy. The authors curate CodeJudge-17K, a high-quality, reasoning-path-annotated dataset, and demonstrate CODE-DiTing’s superior performance over state-of-the-art methods while showing robustness to preference leakage. The approach offers scalable, explainable code evaluation and provides a foundation for future integration with reinforcement learning to further improve code generation systems.

Abstract

Trustworthy evaluation methods for code snippets play a crucial role in neural code generation. Traditional methods, which either rely on reference solutions or require executable test cases, have inherent limitation in flexibility and scalability. The recent LLM-as-Judge methodology offers a promising alternative by directly evaluating functional consistency between the problem description and the generated code. To systematically understand the landscape of these LLM-as-Judge methods, we conduct a comprehensive empirical study across three diverse datasets. Our investigation reveals the pros and cons of two categories of LLM-as-Judge methods: the methods based on general foundation models can achieve good performance but require complex prompts and lack explainability, while the methods based on reasoning foundation models provide better explainability with simpler prompts but demand substantial computational resources due to their large parameter sizes. To address these limitations, we propose CODE-DITING, a novel code evaluation method that balances accuracy, efficiency and explainability. We develop a data distillation framework that effectively transfers reasoning capabilities from DeepSeek-R1671B to our CODE-DITING 1.5B and 7B models, significantly enhancing evaluation explainability and reducing the computational cost. With the majority vote strategy in the inference process, CODE-DITING 1.5B outperforms all models with the same magnitude of parameters and achieves performance which would normally exhibit in a model with 5 times of parameter scale. CODE-DITING 7B surpasses GPT-4o and DeepSeek-V3 671B, even though it only uses 1% of the parameter volume of these large models. Further experiments show that CODEDITING is robust to preference leakage and can serve as a promising alternative for code evaluation.

Figures (4)

• Figure 1: The overall method of Code-DiTing.
• Figure 2: Ablation Study (F1 Score) of Data Filtering Component
• Figure 3: Ablation Study (F1 Score) of PiSSA Component
• Figure 4: Ablation study (F1 Score) of the inference component