Cross-Domain Evaluation of Transformer-Based Vulnerability Detection on Open & Industry Data

TL;DR

The paper examines whether transformer-based vulnerability detection, specifically CodeBERT, can be effectively transferred from open-source to industrial software and integrated into a CI/CD workflow. It builds three cross-domain datasets (GOD, TOD, ID), fine-tunes CodeBERT with multiple balancing strategies, and evaluates cross-domain performance, revealing domain-specific losses and gains. It then implements AI-DO, a GitHub Action-based recommender that localizes vulnerabilities during code review, and validates it through a case study and practitioner survey. The findings demonstrate that technology-agnostic open data with undersampling can improve industrial vulnerability detection, while in-domain industrial training yields the best in-domain results; the work provides practical guidance for shifting vulnerability detection left in industry and outlines avenues for broader language support and multi-annotation datasets.

Abstract

Deep learning solutions for vulnerability detection proposed in academic research are not always accessible to developers, and their applicability in industrial settings is rarely addressed. Transferring such technologies from academia to industry presents challenges related to trustworthiness, legacy systems, limited digital literacy, and the gap between academic and industrial expertise. For deep learning in particular, performance and integration into existing workflows are additional concerns. In this work, we first evaluate the performance of CodeBERT for detecting vulnerable functions in industrial and open-source software. We analyse its cross-domain generalisation when fine-tuned on open-source data and tested on industrial data, and vice versa, also exploring strategies for handling class imbalance. Based on these results, we develop AI-DO(Automating vulnerability detection Integration for Developers' Operations), a Continuous Integration-Continuous Deployment (CI/CD)-integrated recommender system that uses fine-tuned CodeBERT to detect and localise vulnerabilities during code review without disrupting workflows. Finally, we assess the tool's perceived usefulness through a survey with the company's IT professionals. Our results show that models trained on industrial data detect vulnerabilities accurately within the same domain but lose performance on open-source code, while a deep learner fine-tuned on open data, with appropriate undersampling techniques, improves the detection of vulnerabilities.

Figures (3)

• Figure 1: Creation of an annotated dataset by data fusion of two annotation types from static analysers for 40 open source and 8 industry projects.
• Figure 2: Testing performance of CodeBERT fine-tuned on different datasets and with different balancing strategies.
• Figure 3: Flow of the pipeline integrated into a GitHub Action.