Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Analytics of Cybersecurity Crisis Preparation Exercises: What Predicts Success?

Conrad Borchers, Valdemar Švábenský, Sandesh K. Kafle, Kevin K. Tang, Jan Vykopal

Abstract

Instructional alignment, the match between intended cognition and enacted activity, is central to effective instruction but hard to operationalize at scale. We examine alignment in cybersecurity simulations using multimodal traces from 23 teams (76 students) across five exercise sessions. Study 1 codes objectives and team emails with Bloom's taxonomy and models the completion of key exercise tasks with generalized linear mixed models. Alignment, defined as the discrepancy between required and enacted Bloom levels, predicts success, whereas the Bloom category alone does not predict success once discrepancy is considered. Study 2 compares predictive feature families using grouped cross-validation and l1-regularized logistic regression. Text embeddings and log features outperform Bloom-only models (AUC~0.74 and 0.71 vs. 0.55), and their combination performs best (Test AUC~0.80), with Bloom frequencies adding little. Overall, the work offers a measure of alignment for simulations and shows that multimodal traces best forecast performance, while alignment provides interpretable diagnostic insight.

Multimodal Analytics of Cybersecurity Crisis Preparation Exercises: What Predicts Success?

Abstract

Instructional alignment, the match between intended cognition and enacted activity, is central to effective instruction but hard to operationalize at scale. We examine alignment in cybersecurity simulations using multimodal traces from 23 teams (76 students) across five exercise sessions. Study 1 codes objectives and team emails with Bloom's taxonomy and models the completion of key exercise tasks with generalized linear mixed models. Alignment, defined as the discrepancy between required and enacted Bloom levels, predicts success, whereas the Bloom category alone does not predict success once discrepancy is considered. Study 2 compares predictive feature families using grouped cross-validation and l1-regularized logistic regression. Text embeddings and log features outperform Bloom-only models (AUC~0.74 and 0.71 vs. 0.55), and their combination performs best (Test AUC~0.80), with Bloom frequencies adding little. Overall, the work offers a measure of alignment for simulations and shows that multimodal traces best forecast performance, while alignment provides interpretable diagnostic insight.

Paper Structure

This paper contains 25 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Bloom's Taxonomy for the Prediction of Learning Performance
  4. Cybersecurity Education in the AIED Context
  5. Tabletop Exercises (TTXs)
  6. The INJECT Exercise Platform
  7. Educational Goals and Learning Objectives
  8. How Students Learn Using the INJECT Exercise Platform
  9. Methods
  10. Participant Sample
  11. Coding of Emails
  12. Study 1: Explaining Performance via Bloom (RQ1)
  13. Method
  14. Results
  15. Study 2: What Predicts Performance? (RQ2)
  16. ...and 10 more sections

Figures (2)

  • Figure 1: IXP from the student perspective, displaying recent injects.
  • Figure 2: Top ten features in the fully specified model (Text+Log+Bloom). Positive weights indicate higher predicted probability of milestone achievement. Subscripts for embedding-based features represent the n-th dimension of the vector.