Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Empirical assessment of the perception of graphical threat model acceptability

Nathan D. Schiele, Olga Gadyatskaya

TL;DR

This study evaluates the acceptability of three graphical threat models—Attack-Defense Trees, Attack Graphs, and CORAS—among participants with limited technical background using the Method Evaluation Model (MEM). Through a lab experiment with 38 students and a Latin-square design across three cyber threat scenarios, the authors find that ADTs and CORAS are broadly acceptable across tasks, while AGs lag in perceived usefulness, likely due to the absence of dedicated tooling. Tooling availability significantly shapes perceived usefulness and ease of use, suggesting practitioners can achieve acceptable threat modeling outcomes with ADTs or CORAS and that AG tooling should be improved. The work informs practitioners about model selection for non-technical audiences and highlights a clear direction for future research on graphical threat modeling tooling and broader practitioner validation.

Abstract

Threat modeling (TM) is an important aspect of risk analysis and secure software engineering. Graphical threat models are a recommended tool to analyze and communicate threat information. However, the comparison of different graphical threat models, and the acceptability of these threat models for an audience with a limited technical background, is not well understood, despite these users making up a sizable portion of the cybersecurity industry. We seek to compare the acceptability of three general, graphical threat models, Attack-Defense Trees (ADTs), Attack Graphs (AGs), and CORAS, for users with a limited technical background. We conducted a laboratory study with 38 bachelor students who completed tasks with the three threat models across three different scenarios assigned using a Latin square design. Threat model submissions were qualitatively analyzed, and participants filled out a perception questionnaire based on the Method Evaluation Model (MEM). We find that both ADTs and CORAS are broadly acceptable for a wide range of scenarios, and both could be applied successfully by users with a limited technical background; further, we also find that the lack of a specific tool for AGs may have impacted the perceived usefulness of AGs. We can recommend that users with a limited technical background use ADTs or CORAS as a general graphical TM method. Further research on the acceptability of AGs to such an audience and the effect of a dedicated TM tool support is needed.

Empirical assessment of the perception of graphical threat model acceptability

TL;DR

This study evaluates the acceptability of three graphical threat models—Attack-Defense Trees, Attack Graphs, and CORAS—among participants with limited technical background using the Method Evaluation Model (MEM). Through a lab experiment with 38 students and a Latin-square design across three cyber threat scenarios, the authors find that ADTs and CORAS are broadly acceptable across tasks, while AGs lag in perceived usefulness, likely due to the absence of dedicated tooling. Tooling availability significantly shapes perceived usefulness and ease of use, suggesting practitioners can achieve acceptable threat modeling outcomes with ADTs or CORAS and that AG tooling should be improved. The work informs practitioners about model selection for non-technical audiences and highlights a clear direction for future research on graphical threat modeling tooling and broader practitioner validation.

Abstract

Threat modeling (TM) is an important aspect of risk analysis and secure software engineering. Graphical threat models are a recommended tool to analyze and communicate threat information. However, the comparison of different graphical threat models, and the acceptability of these threat models for an audience with a limited technical background, is not well understood, despite these users making up a sizable portion of the cybersecurity industry. We seek to compare the acceptability of three general, graphical threat models, Attack-Defense Trees (ADTs), Attack Graphs (AGs), and CORAS, for users with a limited technical background. We conducted a laboratory study with 38 bachelor students who completed tasks with the three threat models across three different scenarios assigned using a Latin square design. Threat model submissions were qualitatively analyzed, and participants filled out a perception questionnaire based on the Method Evaluation Model (MEM). We find that both ADTs and CORAS are broadly acceptable for a wide range of scenarios, and both could be applied successfully by users with a limited technical background; further, we also find that the lack of a specific tool for AGs may have impacted the perceived usefulness of AGs. We can recommend that users with a limited technical background use ADTs or CORAS as a general graphical TM method. Further research on the acceptability of AGs to such an audience and the effect of a dedicated TM tool support is needed.

Paper Structure

This paper contains 29 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Data Availability
  3. Research Objectives
  4. Background
  5. Related Work
  6. Research Method
  7. Research Method
  8. Theoretical Framework
  9. Study Design
  10. Models and Scenarios Selected
  11. Data Processing and Analysis
  12. Participants and Training
  13. Ethical Considerations
  14. Results
  15. $\text{H}_\text{1}$: Absolute Acceptability (AA) of Threat Models
  16. ...and 14 more sections

Figures (6)

  • Figure 1: The Method Evaluation Model (MEM) moodyMethodEvaluationModel2003.
  • Figure 2: Histogram of last digits of participant student numbers.
  • Figure 3: Distribution of responses to the Likert statement "I find this model to be an effective threat model" (L1) per threat model given as a percentage of the total number of responses for each model.
  • Figure 4: Rankings of the perceived effectiveness of the threat models
  • Figure 5: Actual efficiency metrics per scenario per model
  • ...and 1 more figures