Bridging Expertise Gaps: The Role of LLMs in Human-AI Collaboration for Cybersecurity

TL;DR

This work investigates whether large language models can act as intelligent collaborators to bridge cybersecurity expertise gaps, evaluating two tasks with distinct data modalities—phishing email detection (textual) and intrusion detection (tabular). Through a mixed-methods, within-subjects study (n=58) that compares independent versus AI-assisted performance, the authors show consistent performance gains from collaboration, with phishing benefiting from improved precision and intrusion from enhanced recall, especially under higher cognitive load. Qualitative analyses reveal that AI response definitiveness, explanation style, and tone significantly shape trust, prompting strategies, and decision revision, underscoring the need for calibrated confidence and actionable explanations. The findings offer design guidance for interpretable, adaptive, and trustworthy human–AI teaming in cybersecurity, demonstrating that LLMs can function as cognitive scaffolds that accelerate learning and support non-experts in reasoning about complex security problems.

Abstract

This study investigates whether large language models (LLMs) can function as intelligent collaborators to bridge expertise gaps in cybersecurity decision-making. We examine two representative tasks-phishing email detection and intrusion detection-that differ in data modality, cognitive complexity, and user familiarity. Through a controlled mixed-methods user study, n = 58 (phishing, n = 34; intrusion, n = 24), we find that human-AI collaboration improves task performance,reducing false positives in phishing detection and false negatives in intrusion detection. A learning effect is also observed when participants transition from collaboration to independent work, suggesting that LLMs can support long-term skill development. Our qualitative analysis shows that interaction dynamics-such as LLM definitiveness, explanation style, and tone-influence user trust, prompting strategies, and decision revision. Users engaged in more analytic questioning and showed greater reliance on LLM feedback in high-complexity settings. These results provide design guidance for building interpretable, adaptive, and trustworthy human-AI teaming systems, and demonstrate that LLMs can meaningfully support non-experts in reasoning through complex cybersecurity problems.

Figures (10)

• Figure 1: Study Design: In format A, participants complete the task independently first, followed by collaboration with AI. In format B, the order is reversed, starting with AI collaboration before working independently. Both formats are followed by a questionnaire.
• Figure 2: Phishing email detection study interface used for human-AI collaboration.
• Figure 3: Intrusion detection study interface used for human-AI collaboration.
• Figure 4: Codebook for user questions in phishing email detection study.
• Figure 5: Codebook for LLM responses in phishing email detection study.