Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications

Stephen Burabari Tete

TL;DR

This paper addresses the cybersecurity risks of large language model (LLM)-powered applications by proposing a unified threat modelling and risk analysis framework that combines STRIDE with DREAD and integrates Shostack's Four Question Framework. It catalogues attack vectors—data poisoning, prompt injection, SQL injection, jailbreaking, compositional injection, and insecure output handling—and offers concrete mitigations tailored to LLM-enabled systems. A core contribution is an end-to-end threat model validated through a case study (LLM-Doctor) demonstrating how the framework identifies, prioritizes, and mitigates threats across the development lifecycle. The work provides actionable guidance to enhance the security, reliability, and trustworthiness of LLM-integrated applications across domains such as search, healthcare, and AI assistants, with emphasis on proactive risk management and ongoing threat adaptation.

Abstract

The advent of Large Language Models (LLMs) has revolutionized various applications by providing advanced natural language processing capabilities. However, this innovation introduces new cybersecurity challenges. This paper explores the threat modeling and risk analysis specifically tailored for LLM-powered applications. Focusing on potential attacks like data poisoning, prompt injection, SQL injection, jailbreaking, and compositional injection, we assess their impact on security and propose mitigation strategies. We introduce a framework combining STRIDE and DREAD methodologies for proactive threat identification and risk assessment. Furthermore, we examine the feasibility of an end-to-end threat model through a case study of a custom-built LLM-powered application. This model follows Shostack's Four Question Framework, adjusted for the unique threats LLMs present. Our goal is to propose measures that enhance the security of these powerful AI tools, thwarting attacks, and ensuring the reliability and integrity of LLM-integrated systems.

Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications

TL;DR

This paper addresses the cybersecurity risks of large language model (LLM)-powered applications by proposing a unified threat modelling and risk analysis framework that combines STRIDE with DREAD and integrates Shostack's Four Question Framework. It catalogues attack vectors—data poisoning, prompt injection, SQL injection, jailbreaking, compositional injection, and insecure output handling—and offers concrete mitigations tailored to LLM-enabled systems. A core contribution is an end-to-end threat model validated through a case study (LLM-Doctor) demonstrating how the framework identifies, prioritizes, and mitigates threats across the development lifecycle. The work provides actionable guidance to enhance the security, reliability, and trustworthiness of LLM-integrated applications across domains such as search, healthcare, and AI assistants, with emphasis on proactive risk management and ongoing threat adaptation.

Abstract

The advent of Large Language Models (LLMs) has revolutionized various applications by providing advanced natural language processing capabilities. However, this innovation introduces new cybersecurity challenges. This paper explores the threat modeling and risk analysis specifically tailored for LLM-powered applications. Focusing on potential attacks like data poisoning, prompt injection, SQL injection, jailbreaking, and compositional injection, we assess their impact on security and propose mitigation strategies. We introduce a framework combining STRIDE and DREAD methodologies for proactive threat identification and risk assessment. Furthermore, we examine the feasibility of an end-to-end threat model through a case study of a custom-built LLM-powered application. This model follows Shostack's Four Question Framework, adjusted for the unique threats LLMs present. Our goal is to propose measures that enhance the security of these powerful AI tools, thwarting attacks, and ensuring the reliability and integrity of LLM-integrated systems.
Paper Structure (23 sections, 4 figures, 2 tables)

This paper contains 23 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. LLM-Powered Applications
  3. The Threats and Risks
  4. Overview of Threat Modelling, and Risk Analysis
  5. Threat Modelling (TM)
  6. TM Framework
  7. Risk Analysis
  8. Potential Attacks on LLM-powered Apps (RQ1)
  9. Data Poisoning Attack
  10. Prompt Injection Attacks
  11. SQL Injection Attacks
  12. Jailbreaking Attacks
  13. Compositional Injection Attacks
  14. Insecure Output Handling
  15. Mitigation to attacks on LLM-powered Apps (RQ2)
  16. ...and 8 more sections

Figures (4)

  • Figure 1: AI Assets
  • Figure 2: LLM-Powered application scheme
  • Figure 3: All recently-published large datasets are vulnerable to split-view poisoning attacks. (Carlini et al., 2023)
  • Figure 4: LLM-Application Threat Model