Improving Code Generation via Small Language Model-as-a-judge
Giuseppe Crupi, Rosalia Tufano, Gabriele Bavota
TL;DR
This paper tackles the cost-performance gap in code generation for domain-specific languages by evaluating small language models (SLMs) as judges of code correctness and by using SLM-based judges to select among multiple candidate solutions generated by other SLMs. It demonstrates that fine-tuning SLMs as correctness judges yields moderate agreement with execution-based results, rivaling the prior RankEF approach and approaching GPT-4.1-mini in reliability. When SLMs are used in teams—one generator plus multiple judges—code generation performance often surpasses larger models while using substantially cheaper hardware, with ten candidate solutions typically providing the best gains. The work indicates a practical, scalable path for enterprises to deploy cost-effective in-house code-recommendation systems and sets the stage for broader generalization to other languages and tasks.
Abstract
Large language models (LLMs) have shown remarkable capabilities in automated code generation. While effective for mainstream languages, they may underperform on less common or domain-specific languages, prompting companies to develop in-house code generators. While open-source models can be trained for this, only LLMs with tens of billions of parameters match the performance of commercial tools, demanding costly training and deployment. Recent work proposed supporting code generation with smaller models (SLMs) by generating multiple candidate solutions and using another SLM to select the most likely correct one. The most recent work in this area is the one by Sun et al. [29] presenting RankEF, a T5 model trained to rank code solutions using both execution-based and non-execution-based information. However, Sun et al. do not assess the T5 ranker's classification accuracy, that is, how often it misjudges correct implementations as incorrect or vice versa, leaving open questions about the reliability of LMs as code correctness judges for other tasks (e.g., automated code review). Moreover, their experiments involve relatively old models, making it unclear the extent to which such a methodology would still help companies in cheaply training their own code generators with performance comparable to those of massive LLMs. We present a study addressing these limitations. We train several state-of-the-art SLMs as code correctness judges and assess their ability to discriminate between correct and wrong implementations. We show that modern SLMs outperform RankEF, even without exploiting execution-based information. When used as code rankers, they achieve higher performance gains than RankEF and perform competitively with LLMs 5-25x larger, at a fraction of the cost.