SecureNet: A Comparative Study of DeBERTa and Large Language Models for Phishing Detection
Sakshi Mahendru, Tejul Pandit
TL;DR
Phishing detection across email, SMS, URLs, and webpages is tackled by a comparative study of DeBERTa V3 and large language models (GPT-4 and Gemini 1.5). The authors assemble a diverse evaluation suite combining the HuggingFace Phishing Dataset, Nazario/Nigeria datasets, and GPT-4–generated synthetic data, and they compare fine-tuned DeBERTa V3 classifiers with LLM prompt-tuned detectors. Key findings show DeBERTa V3 variants achieve strong recall on public data, while LLMs exhibit competitive performance on synthetic data, with DeBERTa_3 providing the best overall balance and DeBERTa’s inference speed advantage. The work highlights the importance of data diversity, careful prompt design, and dataset alignment for robust phishing detection, and it suggests hybrid approaches that leverage both architectures for practical cybersecurity deployments.
Abstract
Phishing, whether through email, SMS, or malicious websites, poses a major threat to organizations by using social engineering to trick users into revealing sensitive information. It not only compromises company's data security but also incurs significant financial losses. In this paper, we investigate whether the remarkable performance of Large Language Models (LLMs) can be leveraged for particular task like text classification, particularly detecting malicious content and compare its results with state-of-the-art Deberta V3 (DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing) model. We systematically assess the potential and limitations of both approaches using comprehensive public datasets comprising diverse data sources such as email, HTML, URL, SMS, and synthetic data generation. Additionally, we demonstrate how LLMs can generate convincing phishing emails, making it harder to spot scams and evaluate the performance of both models in this context. Our study delves further into the challenges encountered by DeBERTa V3 during its training phases, fine-tuning methodology and transfer learning processes. Similarly, we examine the challenges associated with LLMs and assess their respective performance. Among our experimental approaches, the transformer-based DeBERTa method emerged as the most effective, achieving a test dataset (HuggingFace phishing dataset) recall (sensitivity) of 95.17% closely followed by GPT-4 providing a recall of 91.04%. We performed additional experiments with other datasets on the trained DeBERTa V3 model and LLMs like GPT 4 and Gemini 1.5. Based on our findings, we provide valuable insights into the effectiveness and robustness of these advanced language models, offering a detailed comparative analysis that can inform future research efforts in strengthening cybersecurity measures for detecting and mitigating phishing threats.