On the Reliability and Explainability of Language Models for Program Generation

TL;DR

The paper investigates the reliability and explainability of language-model–based program generation across eight pre-trained models and five benchmark datasets. It demonstrates that data duplication and test-set overlaps can substantially inflate reported performance, casting doubt on generalization to novel code transformations. Through model-agnostic explanations (SHAP) and token-importance analysis, it reveals that while models capture code grammar and structure, their outputs often closely mirror inputs, indicating limited generative novelty and robustness to input perturbations. The study provides practical guidelines for rigorous benchmarking, robust evaluation beyond accuracy, and the use of explainability to diagnose and improve neural program generation systems.

Abstract

Recent studies have adopted pre-trained language models, such as CodeT5 and CodeGPT, for automated program generation tasks like code generation, repair, and translation. Numerous language model-based approaches have been proposed and evaluated on various benchmark datasets, demonstrating promising performance. However, there is still uncertainty about the reliability of these models, particularly their realistic ability to consistently transform code sequences. This raises the question: are these techniques sufficiently trustworthy for automated program generation? Consequently, Further research is needed to understand model logic and assess reliability and explainability. To bridge these research gaps, we conduct a thorough empirical study of eight popular language models on five representative datasets to determine the capabilities and limitations of automated program generation approaches. We further employ advanced explainable AI approaches to highlight the tokens that significantly contribute to the code transformation. We discover that state-of-the-art approaches suffer from inappropriate performance evaluation stemming from severe data duplication, causing over-optimistic results. Our explainability analysis reveals that, in various experimental scenarios, language models can recognize code grammar and structural information, but they exhibit limited robustness to changes in input sequences. Overall, more rigorous evaluation approaches and benchmarks are critical to enhance the reliability and explainability of automated program generation moving forward. Our findings provide important guidelines for this goal.

Figures (7)

• Figure 1: An explanation example for code review tasks, illustrating the generation of the "setTitle" token from a given source code sequence.
• Figure 2: Distribution of Test-Training Data Duplication and Model Performance across Dataset
• Figure 3: Distribution of Data Duplication within Testing Sets and Corresponding Model Performance Metrics
• Figure 4: Examples of test instances with duplicated sources and different targets from Android_S.
• Figure 5: Average feature importance of token types highlighted by language models on the studied datasets.