Mechanistic Interpretability of Code Correctness in LLMs via Sparse Autoencoders
Kriz Tahimic, Charibeth Cheng
TL;DR
This work addresses how LLMs determine code correctness by applying sparse autoencoders to decompose final-token activations into interpretable latent directions. It identifies two mechanistic families: detection directions that reliably flag incorrect code and steering directions that can modestly correct errors but risk corrupting correct code, revealing a fundamental tradeoff. The study shows that successful code generation relies more on attending to test cases than problem descriptions, and that code-correctness mechanisms learned during pre-training persist through instruction-tuning. Practically, these insights suggest prompting strategies focused on test examples, using predictor directions as error alarms, and employing selective steering to intervene only when errors are anticipated, thereby mitigating widespread code corruption while enabling targeted corrections.
Abstract
As Large Language Models become integral to software development, with substantial portions of AI-suggested code entering production, understanding their internal correctness mechanisms becomes critical for safe deployment. We apply sparse autoencoders to decompose LLM representations, identifying directions that correspond to code correctness. We select predictor directions using t-statistics and steering directions through separation scores from base model representations, then analyze their mechanistic properties through steering, attention analysis, and weight orthogonalization. We find that code correctness directions in LLMs reliably predict incorrect code, while correction capabilities, though statistically significant, involve tradeoffs between fixing errors and preserving correct code. Mechanistically, successful code generation depends on attending to test cases rather than problem descriptions. Moreover, directions identified in base models retain their effectiveness after instruction-tuning, suggesting code correctness mechanisms learned during pre-training are repurposed during fine-tuning. Our mechanistic insights suggest three practical applications: prompting strategies should prioritize test examples over elaborate problem descriptions, predictor directions can serve as error alarms for developer review, and these same predictors can guide selective steering, intervening only when errors are anticipated to prevent the code corruption from constant steering.