Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

garak: A Framework for Security Probing Large Language Models

Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, Nanna Inie

TL;DR

The paper addresses the need for scalable, structured evaluation of Large Language Model security amid evolving adversarial techniques. It proposes garak, a Python-based four-component framework (Generators, Probes, Detectors, Buffs) augmented with an attack-generation module to systematically discover vulnerabilities through exploratory red-teaming. Key contributions include a flexible architecture, a broad suite of probes (including jailbreaks, prompt injections, and data exfiltration tests), detectors for automatic failure scoring, a reporting pipeline, and an adaptive attack generator that learns from prior hits. The framework aims to inform alignment, policy decisions, and risk assessment in real-world deployments, while remaining open-source to foster collaboration and continuous improvement.

Abstract

As Large Language Models (LLMs) are deployed and integrated into thousands of applications, the need for scalable evaluation of how models respond to adversarial attacks grows rapidly. However, LLM security is a moving target: models produce unpredictable output, are constantly updated, and the potential adversary is highly diverse: anyone with access to the internet and a decent command of natural language. Further, what constitutes a security weak in one context may not be an issue in a different context; one-fits-all guardrails remain theoretical. In this paper, we argue that it is time to rethink what constitutes ``LLM security'', and pursue a holistic approach to LLM security evaluation, where exploration and discovery of issues are central. To this end, this paper introduces garak (Generative AI Red-teaming and Assessment Kit), a framework which can be used to discover and identify vulnerabilities in a target LLM or dialog system. garak probes an LLM in a structured fashion to discover potential vulnerabilities. The outputs of the framework describe a target model's weaknesses, contribute to an informed discussion of what composes vulnerabilities in unique contexts, and can inform alignment and policy discussions for LLM deployment.

garak: A Framework for Security Probing Large Language Models

TL;DR

The paper addresses the need for scalable, structured evaluation of Large Language Model security amid evolving adversarial techniques. It proposes garak, a Python-based four-component framework (Generators, Probes, Detectors, Buffs) augmented with an attack-generation module to systematically discover vulnerabilities through exploratory red-teaming. Key contributions include a flexible architecture, a broad suite of probes (including jailbreaks, prompt injections, and data exfiltration tests), detectors for automatic failure scoring, a reporting pipeline, and an adaptive attack generator that learns from prior hits. The framework aims to inform alignment, policy decisions, and risk assessment in real-world deployments, while remaining open-source to foster collaboration and continuous improvement.

Abstract

As Large Language Models (LLMs) are deployed and integrated into thousands of applications, the need for scalable evaluation of how models respond to adversarial attacks grows rapidly. However, LLM security is a moving target: models produce unpredictable output, are constantly updated, and the potential adversary is highly diverse: anyone with access to the internet and a decent command of natural language. Further, what constitutes a security weak in one context may not be an issue in a different context; one-fits-all guardrails remain theoretical. In this paper, we argue that it is time to rethink what constitutes ``LLM security'', and pursue a holistic approach to LLM security evaluation, where exploration and discovery of issues are central. To this end, this paper introduces garak (Generative AI Red-teaming and Assessment Kit), a framework which can be used to discover and identify vulnerabilities in a target LLM or dialog system. garak probes an LLM in a structured fashion to discover potential vulnerabilities. The outputs of the framework describe a target model's weaknesses, contribute to an informed discussion of what composes vulnerabilities in unique contexts, and can inform alignment and policy discussions for LLM deployment.
Paper Structure (37 sections, 5 figures, 3 tables)

This paper contains 37 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Red teaming
  4. Vulnerabilities and policies
  5. Testing LLM Systems
  6. The garak Framework
  7. Generators
  8. Probes
  9. False claims,
  10. Training Data Replay.
  11. Malware Generation.
  12. Invisible tags,
  13. Misleading Claims.
  14. Prompt Injection.
  15. Data/chat exfiltration.
  16. ...and 22 more sections

Figures (5)

  • Figure 1: The garak architecture. Run configuration determines a set of probes to be used. Each probe interacts with the generator, an abstraction for the target LLM or dialog system. Probes pose prompts to this system in an attempt to elicit insecure responses, and generator responses are recorded. Later, detector(s) relevant to the probe's goals are used to score the generator's results.
  • Figure 2: Examples top-level grouping of probe results using the OWASP Top 10 categories of LLM vulnerability. Different groupins lead to different top level results and different concentrations of failure, so it is important to choose a taxonomy applicable to the target context.
  • Figure 3: atkgen data. A sequence of dialog turn exchanges between a human red teamer and a target model, and how these are grouped for training atkgen models
  • Figure 4: Sample garak run, 1 of 2
  • Figure 5: Sample garak run, 2 of 2