Redundancy and Concept Analysis for Code-trained Language Models

TL;DR

This work tackles the interpretability of code-trained language models by performing a fine-grained neuron-level analysis to understand redundancy and the distribution of human-recognizable concepts in latent code representations. It introduces selectivity-guided probing and a suite of neuron-analysis techniques (ranking, clustering, layer-selection, and CKAs) to identify and prune redundant neurons across seven models and four software engineering tasks. Key findings show that over 95% of neurons can be eliminated without significant loss in accuracy, with certain neuron subsets even improving performance, and that important concepts localize to specific token properties (e.g., NUMBER, IDENTIFIER, STRING) and higher-level abstractions for sentence-level tasks. These insights offer practical guidance for transfer learning, model compression, and the decomposition of neural networks into modular components, advancing both the interpretability and efficiency of code intelligence systems.

Abstract

Code-trained language models have proven to be highly effective for various code intelligence tasks. However, they can be challenging to train and deploy for many software engineering applications due to computational bottlenecks and memory constraints. Implementing effective strategies to address these issues requires a better understanding of these 'black box' models. In this paper, we perform the first neuron-level analysis for source code models to identify \textit{important} neurons within latent representations. We achieve this by eliminating neurons that are highly similar or irrelevant to the given task. This approach helps us understand which neurons and layers can be eliminated (redundancy analysis) and where important code properties are located within the network (concept analysis). Using redundancy analysis, we make observations relevant to knowledge transfer and model optimization applications. We find that over 95\% of the neurons are redundant with respect to our code intelligence tasks and can be eliminated without significant loss in accuracy. We also discover several subsets of neurons that can make predictions with baseline accuracy. Through concept analysis, we explore the traceability and distribution of human-recognizable concepts within latent code representations which could be used to influence model predictions. We trace individual and subsets of important neurons to specific code properties and identify 'number' neurons, 'string' neurons, and higher-level 'text' neurons for token-level tasks and higher-level concepts important for sentence-level downstream tasks. This also helps us understand how decomposable and transferable task-related features are and can help devise better techniques for transfer learning, model compression, and the decomposition of deep neural networks into modules.

Figures (4)

• Figure 1: Workflow diagram
• Figure 2: Control Tasks and Selectivity baseline
• Figure 3: Layer Redundancy Analysis, a: Token Tagging, b: Clone Detection, c: Code Search, d: Defect Detection
• Figure 4: Activated words (Red indicates negative activation values and blue indicates positive activation values)