Leveraging Large Language Models for Cybersecurity Risk Assessment -- A Case from Forestry Cyber-Physical Systems

TL;DR

This study investigates locally hosted LLMs with retrieval-augmented generation to support cybersecurity risk assessment in forestry cyber-physical systems under data-protection constraints. Using a design science approach with 12 domain experts, the authors develop a customized Llama 2–based tool and evaluate its ability to generate initial risk assessments, identify threats, and perform redundancy checks, while highlighting the need for human oversight to ensure accuracy and compliance. Key findings show that LLMs can be a valuable starting point and sanity-check for risk assessments but face trust, completeness, and domain-specificity challenges; experts advocate for an agentic, standards-aligned architecture with transparent provenance. The work contributes practical insights for integrating LLM-based assistants into safety-critical risk assessment workflows and points to future directions, including multi-agent systems and broader applicability across safety- and security-critical domains.

Abstract

In safety-critical software systems, cybersecurity activities become essential, with risk assessment being one of the most critical. In many software teams, cybersecurity experts are either entirely absent or represented by only a small number of specialists. As a result, the workload for these experts becomes high, and software engineers would need to conduct cybersecurity activities themselves. This creates a need for a tool to support cybersecurity experts and engineers in evaluating vulnerabilities and threats during the risk assessment process. This paper explores the potential of leveraging locally hosted large language models (LLMs) with retrieval-augmented generation to support cybersecurity risk assessment in the forestry domain while complying with data protection and privacy requirements that limit external data sharing. We performed a design science study involving 12 experts in interviews, interactive sessions, and a survey within a large-scale project. The results demonstrate that LLMs can assist cybersecurity experts by generating initial risk assessments, identifying threats, and providing redundancy checks. The results also highlight the necessity for human oversight to ensure accuracy and compliance. Despite trust concerns, experts were willing to utilize LLMs in specific evaluation and assistance roles, rather than solely relying on their generative capabilities. This study provides insights that encourage the use of LLM-based agents to support the risk assessment process of cyber-physical systems in safety-critical domains.

Figures (2)

• Figure 1: Our design science research approach
• Figure 2: Perceived fulfillment of selected qualities in generated report.