Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?

Alejandro Velasco, David N. Palacio, Daniel Rodriguez-Cardenas, Denys Poshyvanyk

TL;DR

The paper argues that accuracy alone is insufficient to judge masked language models for code, as syntax rules of programming languages may not be causally learned. It introduces SyntaxEval, a two-part framework that (i) masks tokens according to AST node types from a Python CFG to assess syntactic capabilities and (ii) uses a Structural Causal Model to estimate the Average Treatment Effect $\tau$ of syntactic masking on prediction while controlling for confounders. In a case study with two CodeSearchNet-based encoders on ~8k Python snippets, the authors find high overall token-prediction accuracy but no robust causal evidence that syntactic features improve performance; several node types show negative effects, challenging claims that MLMs fully learn syntax. The work highlights the need for syntax- and semantics-aware evaluation and representations in code-understanding models, and provides open data and code to support further investigation.

Abstract

This paper discusses the limitations of evaluating Masked Language Models (MLMs) in code completion tasks. We highlight that relying on accuracy-based measurements may lead to an overestimation of models' capabilities by neglecting the syntax rules of programming languages. To address these issues, we introduce a technique called SyntaxEval in which Syntactic Capabilities are used to enhance the evaluation of MLMs. SyntaxEval automates the process of masking elements in the model input based on their Abstract Syntax Trees (ASTs). We conducted a case study on two popular MLMs using data from GitHub repositories. Our results showed negative causal effects between the node types and MLMs' accuracy. We conclude that MLMs under study fail to predict some syntactic capabilities.

Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?

TL;DR

The paper argues that accuracy alone is insufficient to judge masked language models for code, as syntax rules of programming languages may not be causally learned. It introduces SyntaxEval, a two-part framework that (i) masks tokens according to AST node types from a Python CFG to assess syntactic capabilities and (ii) uses a Structural Causal Model to estimate the Average Treatment Effect of syntactic masking on prediction while controlling for confounders. In a case study with two CodeSearchNet-based encoders on ~8k Python snippets, the authors find high overall token-prediction accuracy but no robust causal evidence that syntactic features improve performance; several node types show negative effects, challenging claims that MLMs fully learn syntax. The work highlights the need for syntax- and semantics-aware evaluation and representations in code-understanding models, and provides open data and code to support further investigation.

Abstract

This paper discusses the limitations of evaluating Masked Language Models (MLMs) in code completion tasks. We highlight that relying on accuracy-based measurements may lead to an overestimation of models' capabilities by neglecting the syntax rules of programming languages. To address these issues, we introduce a technique called SyntaxEval in which Syntactic Capabilities are used to enhance the evaluation of MLMs. SyntaxEval automates the process of masking elements in the model input based on their Abstract Syntax Trees (ASTs). We conducted a case study on two popular MLMs using data from GitHub repositories. Our results showed negative causal effects between the node types and MLMs' accuracy. We conclude that MLMs under study fail to predict some syntactic capabilities.
Paper Structure (9 sections, 3 figures, 2 tables)

This paper contains 9 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Syntactic Causal Evaluation
  4. Case Study Design
  5. Results & Discussion
  6. \ref{['rq:performance']} Syntactic Capabilities Performance
  7. \ref{['rq:causal']} Causal Evaluation Effect
  8. Conclusion & Future Plans
  9. Acknowledgements

Figures (3)

  • Figure 1: SyntaxEval Process for the identifier AST Node.
  • Figure 2: $T_0$ vs. $T_1$ Local Jaccard for Nodes using $M_1$.
  • Figure 3: Syntactic Capabilities Statistically Learned by $M_1$.