Reducing Hallucinations in LLM-Generated Code via Semantic Triangulation
Yihan Dai, Sijie Liang, Haotian Xu, Peichu Xie, Sergey Mechtaev
TL;DR
This work tackles hallucinations in LLM-generated code by introducing semantic triangulation, which uses a non-semantics-preserving problem transformation $\tau$ and a hyperproperty $\phi$ to create cross-task consistency checks across transformed problem instances. The approach replaces single-solution plurality with a bijective mapping of solution classes and cascaded inverse/enumerator subproblems (e.g., $\text{FWD-INV}$, $\text{FWD-SINV}$, $\text{ENUM-SINV}$) implemented in just-tri-it, and it is supported by theory under a stochastic-parrot model with correlated errors using the rearrangement inequality. Empirically, semantic triangulation yields substantial gains on LiveCodeBench and CodeElo over baselines, enables reliable abstention, and handles inexact problems with multiple non-equivalent solutions, demonstrating practical benefits for automated code generation. The results show that exploiting cross-task consistency via bijective error mapping offers a robust, model-agnostic pathway to reduce code-generation hallucinations in black-box LLM settings.
Abstract
When generating code from natural language prompts, an LLM samples programs from a probability distribution, many of which might be incorrect. Sample consensus techniques - such as majority voting or validation against generated tests or specifications - aim to identify a correct program in the sample or abstain if none is valid. However, existing methods often fail to select a correct solution when its sampling probability is low, or when the problem permits multiple valid but non-equivalent solutions. Additionally, they often fail to abstain when no correct solution is present in the sample. To overcome these limitations, we introduce semantic triangulation, which transforms a programming problem in a way that non-trivially alters its semantics while preserving an exact, verifiable mapping between solutions before and after transformation. We theoretically establish that verifying consistency across such problem transformations increases confidence that generated programs reflect accurate generalization rather than spurious statistical correlations, enabling more reliable sample consensus and abstention. On the LiveCodeBench and CodeElo benchmarks, using GPT-4o and DeepSeek-V3 models, semantic triangulation increases reliability of generated code by 21% compared to the method that selects only high-confidence solutions with the probability threshold 0.5, while being able to pinpoint correct solutions at sampling probabilities as low as 0.14. Apart from that, it is also the only approach to consistently form true consensus on tasks with multiple valid but non-equivalent solutions.