Selecting Cybersecurity Requirements: Effects of LLM Use and Professional Software Development Experience
Damjan Fujs, Damjan Vavpotič, Tomaž Hovelja, Marko Poženel
TL;DR
This study investigates how access to Large Language Models (LLMs) and professional software development experience influence the prioritization of cybersecurity requirements for web applications. It uses a controlled experiment with $N=23$ postgraduate students, applying the MoSCoW method to identify $15$ SRs and eight evaluation criteria, analyzed with non-parametric tests. The main finding is that LLM access did not significantly impact SR prioritization or evaluation, while greater professional experience led to higher UX impact ratings and lower risk estimates, with some differences in estimated cost. These results highlight the enduring role of practitioner experience in cybersecurity decision-making and suggest that LLMs may offer limited benefit for evaluation tasks under the tested conditions, informing future empirical work and tool deployment in software security practice.
Abstract
This study investigates how access to Large Language Models (LLMs) and varying levels of professional software development experience affect the prioritization of cybersecurity requirements for web applications. Twenty-three postgraduate students participated in a research study to prioritize security requirements (SRs) using the MoSCoW method and subsequently rated their proposed solutions against multiple evaluation criteria. We divided participants into two groups (one with and the other without access to LLM support during the task). Results showed no significant differences related to LLM use, suggesting that access to LLMs did not noticeably influence how participants evaluated cybersecurity solutions. However, statistically significant differences emerged between experience groups for certain criteria, such as estimated cost to develop a feature, perceived impact on user experience, and risk assessment related to non-implementation of the proposed feature. Participants with more professional experience tended to provide higher ratings for user experience impact and lower risk estimates.