Neuron-Guided Interpretation of Code LLMs: Where, Why, and How?

TL;DR

The paper analyzes code LLMs at the neuron and layer level to reveal language-specific neurons and language-agnostic concept layers that encode abstract code semantics. Using two models across five programming languages, it shows that a small fraction of neurons specialize per language while middle layers form concept layers that are invariant to lexical and syntactic changes. It then demonstrates practical benefits through neuron-guided fine-tuning for code generation, zero-shot semantic clone detection using concept-layer embeddings, and concept-layer guided transfer for code summarization in low-resource languages. These findings offer a path toward more controllable, transferable, and multilingual code intelligence in LLMs. The work also provides mechanistic methods and benchmarks to evaluate interpretability in code models, bridging programming languages and neural representations.

Abstract

Code language models excel on code intelligence tasks, yet their internal interpretability is underexplored. Existing neuron interpretability techniques from NLP are suboptimal for source code due to programming languages formal, hierarchical, and executable nature. We empirically investigate code LLMs at the neuron level, localizing language-specific neurons (selectively responsive to one language) and concept layers (feed-forward layers encoding language-agnostic code representations). We analyze Llama-3.1-8B and Qwen2.5-Coder-32B on multilingual inputs in C++, Java, Python, Go, and JavaScript, measuring neuron selectivity and layerwise contributions during generation. We find (1) neurons specialized for individual languages alongside a universal subset supporting general-purpose generation; and (2) lower layers mainly encode language-specific syntax, while middle layers capture semantic abstractions shared across languages, emerging as concept layers. We demonstrate utility on three tasks: neuron-guided fine-tuning for code generation, clone detection via concept-layer embeddings, and concept-layer-guided transfer for code summarization, each yielding consistent gains in multilingual settings.

Figures (5)

• Figure 1: The Overall Structure of Our Analysis Framework for Language-Specific Neurons.
• Figure 2: Intra-lingual Semantic Invariance Under Lexical Variations across Model Layers.
• Figure 3: Cross-lingual Semantic Equivalence under Syntactic Variations across Model Layers.
• Figure 4: Accuracy of AST Node Prediction across Model Layers.
• Figure 5: F1-scores for Code Clone Detection using Embeddings across Model Layers.