DeepCodeProbe: Towards Understanding What Models Trained on Code Learn

TL;DR

DeepCodeProbe tackles the interpretability gap in code-trained ML models by probing whether non-transformer, small-scale models learn programming-language syntax. Building on AST-Probe, it derives $<d, c, u>$ representations directly from ASTs/CFGs and evaluates embeddings from several code-models across CCD and code summarization tasks. Results show that small models rarely capture full syntax, though they learn meaningful abstractions and task-relevant patterns; scaling up capacity yields diminishing returns and can cause overfitting. The work provides best-practice guidelines for training code-based models and offers an open replication package to apply the probing framework to other models, advancing reliable, efficient software maintenance AI.

Abstract

Machine learning models trained on code and related artifacts offer valuable support for software maintenance but suffer from interpretability issues due to their complex internal variables. These concerns are particularly significant in safety-critical applications where the models' decision-making processes must be reliable. The specific features and representations learned by these models remain unclear, adding to the hesitancy in adopting them widely. To address these challenges, we introduce DeepCodeProbe, a probing approach that examines the syntax and representation learning abilities of ML models designed for software maintenance tasks. Our study applies DeepCodeProbe to state-of-the-art models for code clone detection, code summarization, and comment generation. Findings reveal that while small models capture abstract syntactic representations, their ability to fully grasp programming language syntax is limited. Increasing model capacity improves syntax learning but introduces trade-offs such as increased training time and overfitting. DeepCodeProbe also identifies specific code patterns the models learn from their training data. Additionally, we provide best practices for training models on code to enhance performance and interpretability, supported by an open-source replication package for broader application of DeepCodeProbe in interpreting other code-related models.

Figures (6)

• Figure 1: An example of a Python script and its corresponding AST.
• Figure 2: Annotated AST
• Figure 3: Model architecture for AST-NN zhang2019novel
• Figure 4: Model architecture for FuncGNN nair2020funcgnn
• Figure 5: Model architecture for SummarizationTF shido2019automatic