The Effects of Group Discussion and Role-playing Training on Self-efficacy, Support-seeking, and Reporting Phishing Emails: Evidence from a Mixed-design Experiment

TL;DR

This study evaluates two interactive anti-phishing trainings—group discussion and role-playing—against a no-intervention control in a field setting with N = 105 participants. Using a mixed-design design with three time points and in-situ phishing tests, both trainings improved anti-phishing self-efficacy and support-seeking, while role-playing uniquely enhanced support-seeking when compared to control. Behavioral outcomes showed increased reporting of phishing emails after training, supporting the value of social interaction and experiential learning for organizational phishing resilience. The findings advocate for incorporating interactive, context-rich training into security programs and introduce support-seeking intention as a meaningful metric for evaluating phishing interventions.

Abstract

Organizations rely on phishing interventions to enhance employees' vigilance and safe responses to phishing emails that bypass technical solutions. While various resources are available to counteract phishing, studies emphasize the need for interactive and practical training approaches. To investigate the effectiveness of such an approach, we developed and delivered two anti-phishing trainings, group discussion and role-playing, at a European university. We conducted a pre-registered experiment (N = 105), incorporating repeated measures at three time points, a control group, and three in-situ phishing tests. Both trainings enhanced employees' anti-phishing self-efficacy and support-seeking intention in within-group analyses. Only the role-playing training significantly improved support-seeking intention when compared to the control group. Participants in both trainings reported more phishing tests and demonstrated heightened vigilance to phishing attacks compared to the control group. We discuss practical implications for evaluating and improving phishing interventions and promoting safe responses to phishing threats within organizations.

Figures (7)

• Figure 1: Kruskal–Wallis test of support-seeking deltas.
• Figure 2: Kruskal–Wallis test of self-efficacy deltas.
• Figure 3: Number of Participants (N) mentioned specific counter-phishing practices across all questionnaires. To enable comparison between the control group and the treatment groups, in each cell, a participant is only counted once, even if they mentioned a topic in multiple questionnaires. Bright red indicates the largest number of the row.
• Figure 4: Box plot of self-efficacy scores.
• Figure 5: Box plot of support-seeking scores.