"False negative -- that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing

TL;DR

This paper investigates how developers, across diverse domains, perceive, select, and use static analysis-based security testing (SAST) tools. Through $n=39$ survey responses and $n=20$ in-depth interviews analyzed via reflexive thematic analysis, it reveals 17 key findings about tool selection, expectations, and flaws, highlighting a paradox: practitioners demand detection of real vulnerabilities yet rely on reputation and convenience to choose tools, with manual reviews compensating for SAST limitations. The study argues that benchmarks are often distrustful and proposes new directions for automated evaluation, improved reporting of false negatives, and closer alignment of SAST design with developers’ needs, aiming to reduce undetected vulnerabilities in practice. Overall, the work informs SAST researchers and industry practitioners about how to improve tool usability, evaluation, and integration into CI/CD pipelines to better support secure software development.

Abstract

The demand for automated security analysis techniques, such as static analysis based security testing (SAST) tools continues to increase. To develop SASTs that are effectively leveraged by developers for finding vulnerabilities, researchers and tool designers must understand how developers perceive, select, and use SASTs, what they expect from the tools, whether they know of the limitations of the tools, and how they address those limitations. This paper describes a qualitative study that explores the assumptions, expectations, beliefs, and challenges experienced by developers who use SASTs. We perform in-depth, semi-structured interviews with 20 practitioners who possess a diverse range of software development expertise, as well as a variety of unique security, product, and organizational backgrounds. We identify $17$ key findings that shed light on developer perceptions and desires related to SASTs, and also expose gaps in the status quo - challenging long-held beliefs in SAST design priorities. Finally, we provide concrete future directions for researchers and practitioners rooted in an analysis of our findings.

Figures (1)

• Figure 1: Overview of Semi-structured Interview Guide. We used Laddering technique to delve deeper in each topic, while the semi-structured approach helped us to freely deviate as necessary based on participant response.