Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diagnosing Generalization Failures in Fine-Tuned LLMs: A Cross-Architectural Study on Phishing Detection

Frank Bobe, Gregory D. Vetaw, Chase Pavlick, Darshan Bryner, Matthew Cook, Jose Salas-Vernis

TL;DR

This work tackles the brittleness of fine-tuned LLMs in phishing detection by conducting a cross-architectural study across Llama 3.1 8B, Gemma 2 9B, and Mistral 12B using QLoRA adapters on Enron, SpamAssassin, and Modern Phishing datasets. Through standard and Chain-of-Thought fine-tuning, along with SHAP explanations and mechanistic attention-head analysis, the authors diagnose how architecture, data diversity, and training strategy interact to influence generalization. They reveal key findings: generalization is driven by architecture-data synergy, is architecture-dependent (e.g., Gemma thrives with diverse data while Llama struggles to generalize), and some models (Mistral) show robustness across paradigms. The study uncovers significant data-quality issues (label noise 17-22% and adversarial noise) that confound evaluation and contribute to brittle heuristics, and it provides a practical diagnostic framework and future directions for data curation, curriculum-based training, and architectural exploration to improve real-world robustness in high-stakes AI systems.

Abstract

The practice of fine-tuning Large Language Models (LLMs) has achieved state-of-the-art performance on specialized tasks, yet diagnosing why these models become brittle and fail to generalize remains a critical open problem. To address this, we introduce and apply a multi-layered diagnostic framework to a cross-architectural study. We fine-tune Llama 3.1 8B, Gemma 2 9B, and Mistral models on a high-stakes phishing detection task and use SHAP analysis and mechanistic interpretability to uncover the root causes of their generalization failures. Our investigation reveals three critical findings: (1) Generalization is driven by a powerful synergy between architecture and data diversity. The Gemma 2 9B model achieves state-of-the-art performance (>91\% F1), but only when trained on a stylistically diverse ``generalist'' dataset. (2) Generalization is highly architecture-dependent. We diagnose a specific failure mode in Llama 3.1 8B, which performs well on a narrow domain but cannot integrate diverse data, leading to a significant performance drop. (3) Some architectures are inherently more generalizable. The Mistral model proves to be a consistent and resilient performer across multiple training paradigms. By pinpointing the flawed heuristics responsible for these failures, our work provides a concrete methodology for diagnosing and understanding generalization failures, underscoring that reliable AI requires deep validation of the interplay between architecture, data, and training strategy.

Diagnosing Generalization Failures in Fine-Tuned LLMs: A Cross-Architectural Study on Phishing Detection

TL;DR

This work tackles the brittleness of fine-tuned LLMs in phishing detection by conducting a cross-architectural study across Llama 3.1 8B, Gemma 2 9B, and Mistral 12B using QLoRA adapters on Enron, SpamAssassin, and Modern Phishing datasets. Through standard and Chain-of-Thought fine-tuning, along with SHAP explanations and mechanistic attention-head analysis, the authors diagnose how architecture, data diversity, and training strategy interact to influence generalization. They reveal key findings: generalization is driven by architecture-data synergy, is architecture-dependent (e.g., Gemma thrives with diverse data while Llama struggles to generalize), and some models (Mistral) show robustness across paradigms. The study uncovers significant data-quality issues (label noise 17-22% and adversarial noise) that confound evaluation and contribute to brittle heuristics, and it provides a practical diagnostic framework and future directions for data curation, curriculum-based training, and architectural exploration to improve real-world robustness in high-stakes AI systems.

Abstract

The practice of fine-tuning Large Language Models (LLMs) has achieved state-of-the-art performance on specialized tasks, yet diagnosing why these models become brittle and fail to generalize remains a critical open problem. To address this, we introduce and apply a multi-layered diagnostic framework to a cross-architectural study. We fine-tune Llama 3.1 8B, Gemma 2 9B, and Mistral models on a high-stakes phishing detection task and use SHAP analysis and mechanistic interpretability to uncover the root causes of their generalization failures. Our investigation reveals three critical findings: (1) Generalization is driven by a powerful synergy between architecture and data diversity. The Gemma 2 9B model achieves state-of-the-art performance (>91\% F1), but only when trained on a stylistically diverse ``generalist'' dataset. (2) Generalization is highly architecture-dependent. We diagnose a specific failure mode in Llama 3.1 8B, which performs well on a narrow domain but cannot integrate diverse data, leading to a significant performance drop. (3) Some architectures are inherently more generalizable. The Mistral model proves to be a consistent and resilient performer across multiple training paradigms. By pinpointing the flawed heuristics responsible for these failures, our work provides a concrete methodology for diagnosing and understanding generalization failures, underscoring that reliable AI requires deep validation of the interplay between architecture, data, and training strategy.
Paper Structure (31 sections, 2 equations, 5 figures, 6 tables)

This paper contains 31 sections, 2 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Large Language Models for Cybersecurity
  3. Generalization of Fine-Tuned Models
  4. Interpretability of Fine-Tuned Models
  5. Methodology
  6. Base Models and Fine-Tuning
  7. Datasets and Experimental Design
  8. Experiment 1: Standard Supervised Fine-Tuning
  9. Experiment 2: Chain-of-Thought (CoT) Fine-Tuning
  10. Chain-of-Thought (CoT) Criteria for Phishing Detection
  11. Hold-Out Evaluation Sets
  12. Implementation and Hyperparameter Tuning
  13. Evaluation and Interpretability
  14. Experimental Results
  15. Justification of Hyperparameter Configuration
  16. ...and 16 more sections

Figures (5)

  • Figure 1: SHAP analysis for a false negative prediction on a professional newsletter. Blue tokens push the prediction towards LEGIT, while red pushes towards SPAM. The intensity of the color corresponds to the magnitude of each token's impact; to maintain clarity, the plot only visualizes the most influential tokens. Less impactful words are rendered in lighter shades and may not be visible. The model's output score is displayed at the top; for our binary classifier, a score approaching 1.0 indicates a SPAM prediction, while a lower score indicates a LEGIT prediction.
  • Figure 2: SHAP analysis for a mislabeled MLM scam. The model correctly classifies it as SPAM, driven by strong negative attributions on classic scam keywords.
  • Figure 3: SHAP analysis for a mislabeled Nigerian Prince scam identified within the training set during the data quality audit. Although the model was trained on this specific example using the incorrect label, it correctly classifies the email, demonstrating its ability to override label noise.
  • Figure 4: Specialized attention heads for detecting phishing cues. Top (a): A circuit connecting a subject's call to action to a generic greeting. Bottom (b): A circuit specialized in identifying the grammatical structure of threatening language.
  • Figure 5: Attention pattern for the "Suspicious Link Detector" (Layer 2, Head 15). The head has learned to recognize the specific pattern of an IP address in a URL and link it to the call-to-action.