MulVuln: Enhancing Pre-trained LMs with Shared and Language-Specific Knowledge for Multilingual Vulnerability Detection

TL;DR

MulVuln tackles multilingual vulnerability detection by combining a pre-trained language model encoder that captures shared cross-language patterns with a language-specific parameter pool that injects per-language cues. It dynamically selects or masks language-specific parameters during input embedding, enabling robust cross-language generalization while preserving language-specific knowledge. Evaluated on the REEF dataset, MulVuln achieves state-of-the-art F1-scores (e.g., 72.20% with language-aware masking) and recall near 97%, outperforming 13 baselines by up to 23.59% in F1. The framework provides a practical approach for multilingual SVD and offers insights into when to share representations versus specialize parameters for cross-language code understanding.

Abstract

Software vulnerabilities (SVs) pose a critical threat to safety-critical systems, driving the adoption of AI-based approaches such as machine learning and deep learning for software vulnerability detection. Despite promising results, most existing methods are limited to a single programming language. This is problematic given the multilingual nature of modern software, which is often complex and written in multiple languages. Current approaches often face challenges in capturing both shared and language-specific knowledge of source code, which can limit their performance on diverse programming languages and real-world codebases. To address this gap, we propose MULVULN, a novel multilingual vulnerability detection approach that learns from source code across multiple languages. MULVULN captures both the shared knowledge that generalizes across languages and the language-specific knowledge that reflects unique coding conventions. By integrating these aspects, it achieves more robust and effective detection of vulnerabilities in real-world multilingual software systems. The rigorous and extensive experiments on the real-world and diverse REEF dataset, consisting of 4,466 CVEs with 30,987 patches across seven programming languages, demonstrate the superiority of MULVULN over thirteen effective and state-of-the-art baselines. Notably, MULVULN achieves substantially higher F1-score, with improvements ranging from 1.45% to 23.59% compared to the baseline methods.

Figures (3)

• Figure 1: Overview of MulVuln for multilingual vulnerability detection by enhancing a PLM (e.g., the encoder of CodeT5 including $f_{plm_{e}}(.)$ and $f_{plm_{mha}}(.)$) with shared and language-specific knowledge. For each input $X$, basically, a single parameter matrix $P_X \in \mathbf{R}^{L_p \times D}$ is selected from the parameter pool $\mathcal{P}$ to form the adapted input embedding $X_p = \mathrm{concat}(P_X, X_e)$, encoding both shared and language-specific information. By default, we use the [CLS] token representation for the query function, and the classifier input aggregates the multi-head attention outputs corresponding to the tokens in the selected parameter matrix using mean pooling.
• Figure 2: Visualization of the parameter pool and queries using t-SNE under the Parameter Selection via Key--Parameter Query mechanism (Eq. (\ref{['eq:px']})). Each subplot corresponds to a different programming language. The $\bm{\times}$ marker represents the parameter, and scatter points are queries from test samples of each language. Arrows indicate instance-wise key–parameter associations. Queries radiate outward from their parameter, forming “peacock tail” patterns that reflect sample-level diversity while maintaining a stable language-specific reference.
• Figure 3: Visualization of the parameter pool and queries using t-SNE under the Language-Aware Parameter Masking strategy (Eq. (\ref{['eq:pxmask']})). Each subplot corresponds to a different programming language. Queries remain anchored to their parameter, generally oriented toward it while preserving distinctions between individual samples. For C#, the parameter is positioned farther from its queries, probably due to limited training samples (around 341 versus thousands for other languages), illustrating weaker parameter–query alignment despite cosine similarity-based selection.