White-Basilisk: A Hybrid Model for Code Vulnerability Detection
Ioannis Lamprou, Alexander Shevtsov, Ioannis Arapakis, Sotiris Ioannidis
TL;DR
This work tackles code vulnerability detection under long-context constraints by challenging the notion that bigger models are always better. It introduces White-Basilisk, a compact 200M-parameter model that fuses Mamba state-space layers, Basilisk Self-Attention with linear scaling, and Mixture of Experts to process extremely long code sequences efficiently. Across five benchmarks, including PRIMEVUL and BigVul, the approach achieves state-of-the-art results and demonstrates strong cross-dataset generalization, outperforming much larger models. The results emphasize that architectural innovation enabling long-range context can yield substantial practical gains for domain-specific security tasks, with implications for efficient AI deployment in resource-constrained settings.
Abstract
The proliferation of software vulnerabilities presents a significant challenge to cybersecurity, necessitating more effective detection methodologies. We introduce White-Basilisk, a novel approach to vulnerability detection that demonstrates superior performance while challenging prevailing assumptions in AI model scaling. Utilizing an innovative architecture that integrates Mamba layers, linear self-attention, and a Mixture of Experts framework, White-Basilisk achieves state-of-the-art results in vulnerability detection tasks with a parameter count of only 200M. The model's capacity to process sequences of unprecedented length enables comprehensive analysis of extensive codebases in a single pass, surpassing the context limitations of current Large Language Models (LLMs). White-Basilisk exhibits robust performance on imbalanced, real-world datasets, while maintaining computational efficiency that facilitates deployment across diverse organizational scales. This research not only establishes new benchmarks in code security but also provides empirical evidence that compact, efficiently designed models can outperform larger counterparts in specialized tasks, potentially redefining optimization strategies in AI development for domain-specific applications.