Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?

Sarthak Munshi, Swapnil Pathak, Sonam Ghatode, Thenuga Priyadarshini, Dhivya Chandramouleeswaran, Ashutosh Rana

TL;DR

This work tackles the lack of benchmarks for evaluating LLMs in architectural cloud threat modeling. It introduces ACSE-Eval, a dataset of 100 production AWS deployment scenarios with Diagrams-as-Code, AWS CDK IaC, and threat models mapped to STRIDE, ATT&CK, and OWASP Top 10, alongside a multimodal evaluation framework. Through 0-shot and few-shot assessments of six LLMs, the study finds that GPT-4.1 excels in structured threat identification under few-shot prompts, while Gemini 2.5 Pro shines in 0-shot settings, and Claude 3.7 Sonnet yields strong semantic modeling but weaker threat categorization. The open-source dataset, evaluation toolkit, and methodologies pave the way for reproducible research and highlight avenues like cross-cloud coverage, multimodal context, and interactive threat modeling to advance automated cloud security analysis.

Abstract

While Large Language Models have shown promise in cybersecurity applications, their effectiveness in identifying security threats within cloud deployments remains unexplored. This paper introduces AWS Cloud Security Engineering Eval, a novel dataset for evaluating LLMs cloud security threat modeling capabilities. ACSE-Eval contains 100 production grade AWS deployment scenarios, each featuring detailed architectural specifications, Infrastructure as Code implementations, documented security vulnerabilities, and associated threat modeling parameters. Our dataset enables systemic assessment of LLMs abilities to identify security risks, analyze attack vectors, and propose mitigation strategies in cloud environments. Our evaluations on ACSE-Eval demonstrate that GPT 4.1 and Gemini 2.5 Pro excel at threat identification, with Gemini 2.5 Pro performing optimally in 0-shot scenarios and GPT 4.1 showing superior results in few-shot settings. While GPT 4.1 maintains a slight overall performance advantage, Claude 3.7 Sonnet generates the most semantically sophisticated threat models but struggles with threat categorization and generalization. To promote reproducibility and advance research in automated cybersecurity threat analysis, we open-source our dataset, evaluation metrics, and methodologies.

ACSE-Eval: Can LLMs threat model real-world cloud infrastructure?

TL;DR

This work tackles the lack of benchmarks for evaluating LLMs in architectural cloud threat modeling. It introduces ACSE-Eval, a dataset of 100 production AWS deployment scenarios with Diagrams-as-Code, AWS CDK IaC, and threat models mapped to STRIDE, ATT&CK, and OWASP Top 10, alongside a multimodal evaluation framework. Through 0-shot and few-shot assessments of six LLMs, the study finds that GPT-4.1 excels in structured threat identification under few-shot prompts, while Gemini 2.5 Pro shines in 0-shot settings, and Claude 3.7 Sonnet yields strong semantic modeling but weaker threat categorization. The open-source dataset, evaluation toolkit, and methodologies pave the way for reproducible research and highlight avenues like cross-cloud coverage, multimodal context, and interactive threat modeling to advance automated cloud security analysis.

Abstract

While Large Language Models have shown promise in cybersecurity applications, their effectiveness in identifying security threats within cloud deployments remains unexplored. This paper introduces AWS Cloud Security Engineering Eval, a novel dataset for evaluating LLMs cloud security threat modeling capabilities. ACSE-Eval contains 100 production grade AWS deployment scenarios, each featuring detailed architectural specifications, Infrastructure as Code implementations, documented security vulnerabilities, and associated threat modeling parameters. Our dataset enables systemic assessment of LLMs abilities to identify security risks, analyze attack vectors, and propose mitigation strategies in cloud environments. Our evaluations on ACSE-Eval demonstrate that GPT 4.1 and Gemini 2.5 Pro excel at threat identification, with Gemini 2.5 Pro performing optimally in 0-shot scenarios and GPT 4.1 showing superior results in few-shot settings. While GPT 4.1 maintains a slight overall performance advantage, Claude 3.7 Sonnet generates the most semantically sophisticated threat models but struggles with threat categorization and generalization. To promote reproducibility and advance research in automated cybersecurity threat analysis, we open-source our dataset, evaluation metrics, and methodologies.
Paper Structure (14 sections, 2 equations, 5 figures, 7 tables)

This paper contains 14 sections, 2 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methodology and dataset
  3. Dataset Generation Workflow
  4. Dataset Overview
  5. Experiments
  6. Implementation and Tooling
  7. Evaluating LLMs performance on ACSE-Eval
  8. Framework Coverage
  9. Text Similarity
  10. Comprehensiveness
  11. Discussion
  12. Limitations and Future Direction
  13. Ethical Considerations
  14. Conclusion

Figures (5)

  • Figure 1: Architecture and implementation of a sample cloud infrastructure scenario (a simple S3 pre-signed URL service for image handling). Top: System architecture diagram. Bottom-left: Diagram-as-Code in Python. Bottom-right: AWS CDK IaC in TypeScript.
  • Figure 2: Methodology used for generating the ACSE-Eval dataset.
  • Figure 3: Category-wise breakdown of the AWS services that are part of ACSE-Eval.
  • Figure 4: Distribution of OWASP Top 10 threats in the ACSE-Eval dataset.
  • Figure 5: Heatmaps showing the relationship between threat likelihood and CIA (Confidentiality, Integrity, Availability) impact across all architectures evaluated in ACSE-Eval.