Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking LLAMA Model Security Against OWASP Top 10 For LLM Applications

Nourin Shahin, Izzat Alsmadi

TL;DR

This work benchmarks ten Llama-family models against the OWASP Top 10 for LLM Applications using a 100-prompt adversarial dataset designed per the OWASP AI Testing Guide. It demonstrates a clear inverse relationship between model size and security effectiveness, showing that compact Guard and instruction-tuned variants achieve higher threat detection with lower latency than larger base models. A key contribution is the open-source benchmark dataset and the empirical finding that guard-focused or instruction-tuned, small-footprint models are better suited for production security monitoring. The study also reveals critical gaps, such as System Prompt Leakage and Supply Chain vulnerabilities, underscoring the need for layered defenses and targeted defenses beyond scale alone.

Abstract

As large language models (LLMs) move from research prototypes to enterprise systems, their security vulnerabilities pose serious risks to data privacy and system integrity. This study benchmarks various Llama model variants against the OWASP Top 10 for LLM Applications framework, evaluating threat detection accuracy, response safety, and computational overhead. Using the FABRIC testbed with NVIDIA A30 GPUs, we tested five standard Llama models and five Llama Guard variants on 100 adversarial prompts covering ten vulnerability categories. Our results reveal significant differences in security performance: the compact Llama-Guard-3-1B model achieved the highest detection rate of 76% with minimal latency (0.165s per test), whereas base models such as Llama-3.1-8B failed to detect threats (0% accuracy) despite longer inference times (0.754s). We observe an inverse relationship between model size and security effectiveness, suggesting that smaller, specialized models often outperform larger general-purpose ones in security tasks. Additionally, we provide an open-source benchmark dataset including adversarial prompts, threat labels, and attack metadata to support reproducible research in AI security, [1].

Benchmarking LLAMA Model Security Against OWASP Top 10 For LLM Applications

TL;DR

This work benchmarks ten Llama-family models against the OWASP Top 10 for LLM Applications using a 100-prompt adversarial dataset designed per the OWASP AI Testing Guide. It demonstrates a clear inverse relationship between model size and security effectiveness, showing that compact Guard and instruction-tuned variants achieve higher threat detection with lower latency than larger base models. A key contribution is the open-source benchmark dataset and the empirical finding that guard-focused or instruction-tuned, small-footprint models are better suited for production security monitoring. The study also reveals critical gaps, such as System Prompt Leakage and Supply Chain vulnerabilities, underscoring the need for layered defenses and targeted defenses beyond scale alone.

Abstract

As large language models (LLMs) move from research prototypes to enterprise systems, their security vulnerabilities pose serious risks to data privacy and system integrity. This study benchmarks various Llama model variants against the OWASP Top 10 for LLM Applications framework, evaluating threat detection accuracy, response safety, and computational overhead. Using the FABRIC testbed with NVIDIA A30 GPUs, we tested five standard Llama models and five Llama Guard variants on 100 adversarial prompts covering ten vulnerability categories. Our results reveal significant differences in security performance: the compact Llama-Guard-3-1B model achieved the highest detection rate of 76% with minimal latency (0.165s per test), whereas base models such as Llama-3.1-8B failed to detect threats (0% accuracy) despite longer inference times (0.754s). We observe an inverse relationship between model size and security effectiveness, suggesting that smaller, specialized models often outperform larger general-purpose ones in security tasks. Additionally, we provide an open-source benchmark dataset including adversarial prompts, threat labels, and attack metadata to support reproducible research in AI security, [1].
Paper Structure (28 sections, 2 tables)

This paper contains 28 sections, 2 tables.

Table of Contents

  1. Introduction
  2. Research Objectives
  3. LLMs Evaluation: Dataset Construction
  4. Dataset Design Principles
  5. OWASP Top 10 Coverage.
  6. Related Work and Comparison Study
  7. Prompt Injection and Jailbreak Defenses
  8. LLM Guard Models and Content Moderation
  9. Security Benchmarking and Evaluation Frameworks
  10. Methodology
  11. Experimental Infrastructure
  12. Model Selection and Configuration
  13. Standard Llama Models (5 variants)
  14. Llama Guard Models (5 variants)
  15. Standard vs. Guard Model Output Behavior
  16. ...and 13 more sections