How Dataflow Diagrams Impact Software Security Analysis: an Empirical Experiment

TL;DR

This study empirically evaluates how security-annotated Dataflow Diagrams (DFDs) affect software security analysis performance in a microservice context. Using a within-subjects design with two similar Java apps, it shows a significant 41% improvement in analysis correctness when DFDs are provided, and a 315% boost in evidence correctness when traceability is used. The results also reveal nuanced, task-dependent effects and identify three open challenges (understandability, presentation of missing features, and traceability accessibility). Overall, the work supports adopting model-based, security-annotated architectures while highlighting practical considerations for integrating DFDs into security workflows. This informs practitioners and researchers about the potential benefits and limitations of DFDs in real-world security analyses and certification contexts.

Abstract

Models of software systems are used throughout the software development lifecycle. Dataflow diagrams (DFDs), in particular, are well-established resources for security analysis. Many techniques, such as threat modelling, are based on DFDs of the analysed application. However, their impact on the performance of analysts in a security analysis setting has not been explored before. In this paper, we present the findings of an empirical experiment conducted to investigate this effect. Following a within-groups design, participants were asked to solve security-relevant tasks for a given microservice application. In the control condition, the participants had to examine the source code manually. In the model-supported condition, they were additionally provided a DFD of the analysed application and traceability information linking model items to artefacts in source code. We found that the participants (n = 24) performed significantly better in answering the analysis tasks correctly in the model-supported condition (41% increase in analysis correctness). Further, participants who reported using the provided traceability information performed better in giving evidence for their answers (315% increase in correctness of evidence). Finally, we identified three open challenges of using DFDs for security analysis based on the insights gained in the experiment.

Figures (8)

• Figure 1: Example DFD provided to participants in model-supported condition.
• Figure 2: Screenshots of an example traceability information for the annotation authorization_server (top) and the target of the URL (bottom).
• Figure 3: Participants' (a) programming skills, (b) experience in reading Java code, and (c) work experience as developers. All self-reported.
• Figure 4: Comparison across the two treatments of the participants' average score in analysis correctness. Per task and overall.
• Figure 5: Comparison across the two treatments of the participants' average score in correctness of evidence. Per task and overall.