Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Studying Vulnerable Code Entities in R

Zixiao Zhao, Millon Madhur Das, Fatemeh H. Fard

TL;DR

This work investigates the vulnerability of Code-PLMs for the R language by applying the black-box CodeAttack framework to a large R code-comment dataset, revealing that identifiers are the most vulnerable code entities. The authors fine-tune CodeT5 on 32,671 R functions and demonstrate that adversarial perturbations can significantly alter code summaries, with R showing smaller BLEU declines than PHP, Python, or Java. The findings highlight the importance of token types and code structure in R, offering a first step toward robust, R-specific code-PLMs for tasks such as code summarization and method-name prediction. The study provides open-source scripts to enable replication and future exploration across languages and tasks.

Abstract

Pre-trained Code Language Models (Code-PLMs) have shown many advancements and achieved state-of-the-art results for many software engineering tasks in the past few years. These models are mainly targeted for popular programming languages such as Java and Python, leaving out many other ones like R. Though R has a wide community of developers and users, there is little known about the applicability of Code-PLMs for R. In this preliminary study, we aim to investigate the vulnerability of Code-PLMs for code entities in R. For this purpose, we use an R dataset of code and comment pairs and then apply CodeAttack, a black-box attack model that uses the structure of code to generate adversarial code samples. We investigate how the model can attack different entities in R. This is the first step towards understanding the importance of R token types, compared to popular programming languages (e.g., Java). We limit our study to code summarization. Our results show that the most vulnerable code entity is the identifier, followed by some syntax tokens specific to R. The results can shed light on the importance of token types and help in developing models for code summarization and method name prediction for the R language.

Studying Vulnerable Code Entities in R

TL;DR

This work investigates the vulnerability of Code-PLMs for the R language by applying the black-box CodeAttack framework to a large R code-comment dataset, revealing that identifiers are the most vulnerable code entities. The authors fine-tune CodeT5 on 32,671 R functions and demonstrate that adversarial perturbations can significantly alter code summaries, with R showing smaller BLEU declines than PHP, Python, or Java. The findings highlight the importance of token types and code structure in R, offering a first step toward robust, R-specific code-PLMs for tasks such as code summarization and method-name prediction. The study provides open-source scripts to enable replication and future exploration across languages and tasks.

Abstract

Pre-trained Code Language Models (Code-PLMs) have shown many advancements and achieved state-of-the-art results for many software engineering tasks in the past few years. These models are mainly targeted for popular programming languages such as Java and Python, leaving out many other ones like R. Though R has a wide community of developers and users, there is little known about the applicability of Code-PLMs for R. In this preliminary study, we aim to investigate the vulnerability of Code-PLMs for code entities in R. For this purpose, we use an R dataset of code and comment pairs and then apply CodeAttack, a black-box attack model that uses the structure of code to generate adversarial code samples. We investigate how the model can attack different entities in R. This is the first step towards understanding the importance of R token types, compared to popular programming languages (e.g., Java). We limit our study to code summarization. Our results show that the most vulnerable code entity is the identifier, followed by some syntax tokens specific to R. The results can shed light on the importance of token types and help in developing models for code summarization and method name prediction for the R language.
Paper Structure (16 sections, 3 figures, 2 tables)

This paper contains 16 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Code-PLMs and R Studies
  4. Adversarial Attacks
  5. Adversarial Attacks on Code
  6. CodeAttack
  7. Methodology
  8. Fine-tune a Code-PLM on R
  9. CodeT5
  10. R dataset
  11. Experiment and result
  12. Experiment Setup
  13. CodeAttack Result
  14. Discussions
  15. THREATS TO VALIDITY
  16. ...and 1 more sections

Figures (3)

  • Figure 1: CodeAttack makes a small change to one token in the input code snippet which causes significant changes to the code summary obtained from the SOTA pre-trained programming language models fine-tuned on R.
  • Figure 2: Normalized success rate of different token types in R.
  • Figure 3: The importance of the code entities (i.e., token types).