Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfalls

TL;DR

This paper tackles the unreliability of LLM-based judges in domain-specific code evaluation by creating a COBOL-focused failure taxonomy and a lightweight analytic checker that generates domain-specific hints. By injecting these hints into the judge prompts, the authors demonstrate substantial improvements in error detection from about 45–53% to up to 94% for certain configurations, highlighting the practical value of analytic-guided evaluation in production pipelines. The approach preserves auditability and avoids retraining, and the authors release the dataset, prompts, and checker to enable replication. Overall, the work presents a viable path toward more reliable, hybrid evaluation systems for high-stakes code generation tasks.

Abstract

Large Language Models are increasingly deployed as judges (LaaJ) in code generation pipelines. While attractive for scalability, LaaJs tend to overlook domain specific issues raising concerns about their reliability in critical evaluation tasks. To better understand these limitations in practice, we examine LaaJ behavior in a concrete industrial use case: legacy code modernization via COBOL code generation. In this setting, we find that even production deployed LaaJs can miss domain critical errors, revealing consistent blind spots in their evaluation capabilities. To better understand these blind spots, we analyze generated COBOL programs and associated LaaJs judgments, drawing on expert knowledge to construct a preliminary taxonomy. Based on this taxonomy, we develop a lightweight analytic checker tool that flags over 30 domain specific issues observed in practice. We use its outputs as analytic hints, dynamically injecting them into the judges prompt to encourage LaaJ to revisit aspects it may have overlooked. Experiments on a test set of 100 programs using four production level LaaJs show that LaaJ alone detects only about 45% of the errors present in the code (in all judges we tested), while the analytic checker alone lacks explanatory depth. When combined, the LaaJ+Hints configuration achieves up to 94% coverage (for the best performing judge and injection prompt) and produces qualitatively richer, more accurate explanations, demonstrating that analytic-LLM hybrids can substantially enhance evaluation reliability in deployed pipelines. We release the dataset and all used prompts.

Figures (4)

• Figure 1: Full Workflow for improving LLM-as-a-Judge evaluations. The main pipeline (blue) generates candidate outputs from the LLM and scores them using a LaaJ. Two refinement loops are overlaid: (1) Blind Spots Collection (green), where SMEs analyze LaaJ errors and curate a taxonomy of blind spots, and (2) Hints Optimization (red), where targeted hints are derived and injected via an Analytic Checker to guide LaaJ reasoning. This process allows LaaJ evaluations to be continuously improved by addressing specific weaknesses in model judgment.
• Figure 2: Taxonomy of domain-specific evaluation issues identified in COBOL LaaJ failures.
• Figure 3: Hybrid Laajs detection rates (naive vs optimized hint injection schemes), with Native Laajs detection rates.
• Figure 4: Percent of native judge errors rediscovered by hybrid judges (with best hint-injection prompt).