Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Phishing Email Detection Performance of Small Large Language Models

Zijie Lin, Zikang Liu, Hanbo Fan

TL;DR

This work demonstrates that three-billion-parameter LLMs, once guided by carefully engineered prompts and training with explanations, can achieve competitive phishing email detection performance while offering interpretability and efficiency. By integrating prompt engineering, explanation-augmented fine-tuning with LoRA, and model ensemble, the authors close the gap between small and large LLMs on SpamAssassin and CEAS_08 datasets. The approach yields substantial gains over vanilla prompting, enables strong transferability to unseen datasets, and even sometimes surpasses standard-sized LLMs and traditional ML baselines. The findings suggest practical deployments of cost-effective, explainable phishing detectors that scale better than giant LLMs without sacrificing accuracy.

Abstract

Large language models(LLMs) have demonstrated remarkable performance on many natural language processing(NLP) tasks and have been employed in phishing email detection research. However, in current studies, well-performing LLMs typically contain billions or even tens of billions of parameters, requiring enormous computational resources. To reduce computational costs, we investigated the effectiveness of small-parameter LLMs for phishing email detection. These LLMs have around 3 billion parameters and can run on consumer-grade GPUs. However, small LLMs often perform poorly in phishing email detection task. To address these issues, we designed a set of methods including Prompt Engineering, Explanation Augmented Fine-tuning, and Model Ensemble to improve phishing email detection capabilities of small LLMs. We validated the effectiveness of our approach through experiments, significantly improving both accuracy and F1 score on the SpamAssassin and CEAS\_08 datasets. Furthermore, the fine-tuned models demonstrated strong transferability, achieving robust performance across multiple unseen phishing datasets, outperforming traditional baselines and approaching standard-sized LLMs.

Improving Phishing Email Detection Performance of Small Large Language Models

TL;DR

This work demonstrates that three-billion-parameter LLMs, once guided by carefully engineered prompts and training with explanations, can achieve competitive phishing email detection performance while offering interpretability and efficiency. By integrating prompt engineering, explanation-augmented fine-tuning with LoRA, and model ensemble, the authors close the gap between small and large LLMs on SpamAssassin and CEAS_08 datasets. The approach yields substantial gains over vanilla prompting, enables strong transferability to unseen datasets, and even sometimes surpasses standard-sized LLMs and traditional ML baselines. The findings suggest practical deployments of cost-effective, explainable phishing detectors that scale better than giant LLMs without sacrificing accuracy.

Abstract

Large language models(LLMs) have demonstrated remarkable performance on many natural language processing(NLP) tasks and have been employed in phishing email detection research. However, in current studies, well-performing LLMs typically contain billions or even tens of billions of parameters, requiring enormous computational resources. To reduce computational costs, we investigated the effectiveness of small-parameter LLMs for phishing email detection. These LLMs have around 3 billion parameters and can run on consumer-grade GPUs. However, small LLMs often perform poorly in phishing email detection task. To address these issues, we designed a set of methods including Prompt Engineering, Explanation Augmented Fine-tuning, and Model Ensemble to improve phishing email detection capabilities of small LLMs. We validated the effectiveness of our approach through experiments, significantly improving both accuracy and F1 score on the SpamAssassin and CEAS\_08 datasets. Furthermore, the fine-tuned models demonstrated strong transferability, achieving robust performance across multiple unseen phishing datasets, outperforming traditional baselines and approaching standard-sized LLMs.
Paper Structure (15 sections, 8 equations, 4 figures, 4 tables)

This paper contains 15 sections, 8 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Statement
  5. Prompt Engineering
  6. Explanation Augmented Fine-tuning
  7. Model Ensemble
  8. Experiments
  9. Vanilla Prompt Engineering Performance
  10. Explanation Augmented Fine-tuning Performance
  11. Model Ensemble
  12. Ablation Study
  13. Experiment on Transferability
  14. Limitation
  15. Conclusion

Figures (4)

  • Figure 1: The workflow of improving phishing email detection performance of small LLMs
  • Figure 2: Using GPT-4o-mini to augment the original phishing email dataset with explanation
  • Figure 3: Accuracy and F1 scores of different models evaluated on multiple datasets. Note that small-scale LLMs and traditional machine learning models are specifically trained on the SpamAssassin dataset, while for standard-sized LLMs (e.g., GPT-3.5-Turbo), we employ direct prompting for phishing email detection without additional training.
  • Figure 4: Accuracy and F1 scores of different models evaluated on multiple datasets. Note that small-scale LLMs and traditional machine learning models are specifically trained on the CEAS_08 dataset, while for standard-sized LLMs (e.g., GPT-3.5-Turbo), we employ direct prompting for phishing email detection without additional training.