Exploring Software Naturalness through Neural Language Models

TL;DR

This work tests the Software Naturalness hypothesis by training a BERT-like transformer from scratch on raw C code (C-BERT) and evaluating its ability to learn AST structure and detect vulnerabilities without relying on compiler-derived features. It systematically compares tokenization strategies (Char, KeyChar, SPE) and pre-training with MLM versus Whole Word Masking, demonstrating that AST-like information can be recovered through language models alone. The VI results show that C-BERT, especially with appropriate aggregation and masking, can outperform graph-based GGNN approaches that depend on compiled features, highlighting practical benefits for incomplete or in-development code. The findings suggest that transformer-based LMs on source code can be effectively integrated into software security workflows and development pipelines, with future work aiming to joint-learn AST and VI tasks.

Abstract

The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks. Present approaches to code analysis depend heavily on features derived from the Abstract Syntax Tree (AST) while our transformer-based language models work on raw source code. This work is the first to investigate whether such language models can discover AST features automatically. To achieve this, we introduce a sequence labeling task that directly probes the language models understanding of AST. Our results show that transformer based language models achieve high accuracy in the AST tagging task. Furthermore, we evaluate our model on a software vulnerability identification task. Importantly, we show that our approach obtains vulnerability identification results comparable to graph based approaches that rely heavily on compilers for feature extraction.