Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatment

Neha Nagaraja, Hayretdin Bahsi

TL;DR

This work introduces a goal-driven risk assessment framework for LLM-powered systems that combines system modeling with Attack-Defense Trees (ADTrees) and Common Vulnerability Scoring System (CVSS) based exploitability scoring to support structured, comparable analysis.

Abstract

Large Language Models (LLMs) are increasingly integrated into safety-critical workflows, yet existing security analyses remain fragmented and often isolate model behavior from the broader system context. This work introduces a goal-driven risk assessment framework for LLM-powered systems that combines system modeling with Attack-Defense Trees (ADTrees) and Common Vulnerability Scoring System (CVSS)-based exploitability scoring to support structured, comparable analysis. We demonstrate the framework through a healthcare case study, modeling multi-step attack paths targeting intervention in medical procedures, leakage of electronic health record (EHR) data, and disruption of service availability. Our analysis indicates that threats spanning (i) conventional cyber, (ii) adversarial ML, and (iii) conversational attacks that manipulate prompts or context often consolidate into a small number of dominant paths and shared system choke points, enabling targeted defenses to yield meaningful reductions in path exploitability. By systematically comparing defense portfolios, we align these risks with established vulnerability management practices and provide a domain-agnostic workflow applicable to other LLM-enabled critical systems.

Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatment

TL;DR

This work introduces a goal-driven risk assessment framework for LLM-powered systems that combines system modeling with Attack-Defense Trees (ADTrees) and Common Vulnerability Scoring System (CVSS) based exploitability scoring to support structured, comparable analysis.

Abstract

Large Language Models (LLMs) are increasingly integrated into safety-critical workflows, yet existing security analyses remain fragmented and often isolate model behavior from the broader system context. This work introduces a goal-driven risk assessment framework for LLM-powered systems that combines system modeling with Attack-Defense Trees (ADTrees) and Common Vulnerability Scoring System (CVSS)-based exploitability scoring to support structured, comparable analysis. We demonstrate the framework through a healthcare case study, modeling multi-step attack paths targeting intervention in medical procedures, leakage of electronic health record (EHR) data, and disruption of service availability. Our analysis indicates that threats spanning (i) conventional cyber, (ii) adversarial ML, and (iii) conversational attacks that manipulate prompts or context often consolidate into a small number of dominant paths and shared system choke points, enabling targeted defenses to yield meaningful reductions in path exploitability. By systematically comparing defense portfolios, we align these risks with established vulnerability management practices and provide a domain-agnostic workflow applicable to other LLM-enabled critical systems.
Paper Structure (22 sections, 6 equations, 8 figures, 6 tables)

This paper contains 22 sections, 6 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Methodology
  4. System Modeling
  5. Threat Modeling
  6. Systematic Risk Assessment
  7. Attack–Defense Tree Modelling
  8. CVSS-based Quantification of Attack–Defense Trees
  9. Risk Treatment
  10. Results
  11. RQ1 – Goal-Driven Attack–Defense Trees
  12. G1: Intervening in medical procedures
  13. G2: Leakage of Electronic Health Record (EHR) Data
  14. G3: Disruption of Access to EHR Data
  15. Quantitative ADT Analysis using CVSS Exploitability
  16. ...and 7 more sections

Figures (8)

  • Figure 1: System Architecture of the LLM-based Healthcare Assistant
  • Figure 2: Toy ADT illustrating precondition–execution–impact decomposition and defense placement for G2 (EHR leakage)
  • Figure 3: Toy example showing how we compute a path-level CVSS-style score
  • Figure 4: Toy example for risk treatment
  • Figure 5: Workflow of the LLM-based Healthcare System.
  • ...and 3 more figures