Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Understanding What Code Language Models Learned

Toufique Ahmed, Dian Yu, Chengxuan Huang, Cathy Wang, Prem Devanbu, Kenji Sagae

TL;DR

This work investigates whether code pre-trained language models capture true computational semantics rather than merely lexical patterns by applying meaning-preserving transformations to code and testing masked-token reconstruction. Comparing code-specialized models (CodeBERT, GraphCodeBERT) with a natural-language baseline (RoBERTa), it shows that CodeBERT and GraphCodeBERT maintain high accuracy on original and transformed code, and that semantically equivalent forms cluster in embedding space, indicating semantic understanding. Key findings include robustness to variable renaming, context-length effects emphasizing following tokens, and a measurable drop under more aggressive condition refactoring, all suggesting that PLMs encode meaningful code semantics rather than surface form alone. The results have implications for evaluating code intelligence, supporting the view that PLMs can develop robust semantic representations with practical impact on code understanding and tooling.

Abstract

Pre-trained language models are effective in a variety of natural language tasks, but it has been argued their capabilities fall short of fully learning meaning or understanding language. To understand the extent to which language models can learn some form of meaning, we investigate their ability to capture semantics of code beyond superficial frequency and co-occurrence. In contrast to previous research on probing models for linguistic features, we study pre-trained models in a setting that allows for objective and straightforward evaluation of a model's ability to learn semantics. In this paper, we examine whether such models capture the semantics of code, which is precisely and formally defined. Through experiments involving the manipulation of code fragments, we show that code pre-trained models of code learn a robust representation of the computational semantics of code that goes beyond superficial features of form alone

Towards Understanding What Code Language Models Learned

TL;DR

This work investigates whether code pre-trained language models capture true computational semantics rather than merely lexical patterns by applying meaning-preserving transformations to code and testing masked-token reconstruction. Comparing code-specialized models (CodeBERT, GraphCodeBERT) with a natural-language baseline (RoBERTa), it shows that CodeBERT and GraphCodeBERT maintain high accuracy on original and transformed code, and that semantically equivalent forms cluster in embedding space, indicating semantic understanding. Key findings include robustness to variable renaming, context-length effects emphasizing following tokens, and a measurable drop under more aggressive condition refactoring, all suggesting that PLMs encode meaningful code semantics rather than surface form alone. The results have implications for evaluating code intelligence, supporting the view that PLMs can develop robust semantic representations with practical impact on code understanding and tooling.

Abstract

Pre-trained language models are effective in a variety of natural language tasks, but it has been argued their capabilities fall short of fully learning meaning or understanding language. To understand the extent to which language models can learn some form of meaning, we investigate their ability to capture semantics of code beyond superficial frequency and co-occurrence. In contrast to previous research on probing models for linguistic features, we study pre-trained models in a setting that allows for objective and straightforward evaluation of a model's ability to learn semantics. In this paper, we examine whether such models capture the semantics of code, which is precisely and formally defined. Through experiments involving the manipulation of code fragments, we show that code pre-trained models of code learn a robust representation of the computational semantics of code that goes beyond superficial features of form alone
Paper Structure (28 sections, 7 figures, 9 tables)

This paper contains 28 sections, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Models' Accuracy with Meaning Preserving Transformation
  5. Like Compilers do!
  6. Block Swap
  7. Operand Swap
  8. Robustness & Structure of Semantic Representation
  9. Consistent Variable Renaming
  10. Context length and direction
  11. Refactoring of conditional statement
  12. Distance in embedding space
  13. Experiments and Results
  14. Dataset, Models, and Experiments
  15. Results and Analysis
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Semantically identical forms after transformation: Block Swap. If the block is swapped, the model should predict "==" rather than "!=".
  • Figure 2: Semantically identical forms after transformation: Operand Swap. If the operand is swapped, the model should predict ">=" rather than "<=".
  • Figure 3: Semantically equivalent and non-equivalent pairs of program snippets. We found that the embedding vector distance between the programs in equivalent pairs is lower than the distance between the programs in nonequivalent pairs, even though the programs in the nonequivalent pairs are more similar to each other than the programs in the equivalent pairs from the superficial perspective of tokens and token sequences.
  • Figure 4: Impact of variable renaming on the performance of GraphCodeBERT model in block swap transformation. Results show that the model's performance degrades by less than 1%-4%.
  • Figure 5: Impact of refactoring on the performance of GraphCodeBERT model in block swap transformation. Results show that the model's performance decreases by around 10% with such operation in both original and transformed code, but is still quite good.
  • ...and 2 more figures