Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vul-LMGNNs: Fusing language models and online-distilled graph neural networks for code vulnerability detection

Ruitong Liu, Yanbin Wang, Haitao Xu, Jianguo Sun, Fan Zhang, Peiyue Li, Zhenhao Guo

TL;DR

Vul-LMGNNs is proposed, integrating pre-trained codeLMs with GNNs to enable cross-layer propagation of semantic and structural information, and outperform 17 state-of-the-art approaches on real-world vulnerability datasets.

Abstract

Code Language Models (codeLMs) and Graph Neural Networks (GNNs) are widely used in code vulnerability detection. However, GNNs often rely on aggregating information from adjacent nodes, limiting structural information propagation across layers. While codeLMs can supplement GNNs with semantic information, existing integration methods underexplore their collaborative potential. To address these challenges, we propose Vul-LMGNNs, integrating pre-trained codeLMs with GNNs to enable cross-layer propagation of semantic and structural information. Vul-LMGNNs leverage Code Property Graphs (CPGs) to incorporate syntax, control flow, and data dependencies, using gated GNNs for structural extraction. An online knowledge distillation (KD) mechanism allows a student GNN to capture structural information from a trained counterpart via alternating training. Additionally, an "implicit-explicit" joint training framework leverages codeLMs to initialize embeddings and propagate code semantics. In the explicit phase, it performs late fusion via linear interpolation. Evaluations on real-world vulnerability datasets show Vul-LMGNNs outperform 17 state-of-the-art approaches. Source code is available at: https://github.com/Vul-LMGNN/vul-LMGNN.

Vul-LMGNNs: Fusing language models and online-distilled graph neural networks for code vulnerability detection

TL;DR

Vul-LMGNNs is proposed, integrating pre-trained codeLMs with GNNs to enable cross-layer propagation of semantic and structural information, and outperform 17 state-of-the-art approaches on real-world vulnerability datasets.

Abstract

Code Language Models (codeLMs) and Graph Neural Networks (GNNs) are widely used in code vulnerability detection. However, GNNs often rely on aggregating information from adjacent nodes, limiting structural information propagation across layers. While codeLMs can supplement GNNs with semantic information, existing integration methods underexplore their collaborative potential. To address these challenges, we propose Vul-LMGNNs, integrating pre-trained codeLMs with GNNs to enable cross-layer propagation of semantic and structural information. Vul-LMGNNs leverage Code Property Graphs (CPGs) to incorporate syntax, control flow, and data dependencies, using gated GNNs for structural extraction. An online knowledge distillation (KD) mechanism allows a student GNN to capture structural information from a trained counterpart via alternating training. Additionally, an "implicit-explicit" joint training framework leverages codeLMs to initialize embeddings and propagate code semantics. In the explicit phase, it performs late fusion via linear interpolation. Evaluations on real-world vulnerability datasets show Vul-LMGNNs outperform 17 state-of-the-art approaches. Source code is available at: https://github.com/Vul-LMGNN/vul-LMGNN.
Paper Structure (22 sections, 19 equations, 7 figures, 7 tables, 2 algorithms)

This paper contains 22 sections, 19 equations, 7 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sequence Based Models
  4. Graph Based Models
  5. Vul-LMGNNs
  6. Code Representation
  7. Code Property Graphs
  8. Gated Graph Neural Network
  9. Online Knowledge Distillation for GGNNs
  10. Extracting Code Sequence Semantics By CodeLMs
  11. Joint Training of CodeLM and GNNs
  12. Implicit Stage
  13. Explicit Stage
  14. Dataset Review
  15. Experiments results
  16. ...and 7 more sections

Figures (7)

  • Figure 1: Overview of the Vul-LMGNNs Vulnerability Detection Framework.
  • Figure 2: A CPG for the example source code. Edge-type legend: Blue = AST, Red = CFG, Purple = PDG.
  • Figure 3: Feature extraction and node embedding phases.
  • Figure 4: Schematic diagram of online knowledge distillation based on knowledge alignment.
  • Figure 5: Detection accuracy for the top-30 high-frequency CWE vulnerability types in DiverseVul. Note: CodeBertGGNN represents a basic variant of the proposed Vul-LMGNNs, specifically Vul-LMGNNs using CodeBERT as the Language Model (LM)
  • ...and 2 more figures