Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critique

Suhas Hariharan, Zainab Ali Majid, Jaime Raldua Veuthey, Jacob Haimes

TL;DR

This exploration of Meta's CyberSecEval methodology is used as a test case for LLM-assisted benchmark analysis, and highlights key drawbacks of the methodology.

Abstract

A key development in the cybersecurity evaluations space is the work carried out by Meta, through their CyberSecEval approach. While this work is undoubtedly a useful contribution to a nascent field, there are notable features that limit its utility. Key drawbacks focus on the insecure code detection part of Meta's methodology. We explore these limitations, and use our exploration as a test case for LLM-assisted benchmark analysis.

Rethinking CyberSecEval: An LLM-Aided Approach to Evaluation Critique

TL;DR

This exploration of Meta's CyberSecEval methodology is used as a test case for LLM-assisted benchmark analysis, and highlights key drawbacks of the methodology.

Abstract

A key development in the cybersecurity evaluations space is the work carried out by Meta, through their CyberSecEval approach. While this work is undoubtedly a useful contribution to a nascent field, there are notable features that limit its utility. Key drawbacks focus on the insecure code detection part of Meta's methodology. We explore these limitations, and use our exploration as a test case for LLM-assisted benchmark analysis.

Paper Structure

This paper contains 15 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Critique and Results
  3. ICD: Static Analysis
  4. Instruct: Compliance Issue
  5. Autocomplete: Code Comments and Identifiers
  6. Conclusions
  7. Limitations
  8. Example prompts
  9. Instruct
  10. Autocomplete
  11. Instruct: Compliance Issue Example
  12. Autocomplete: Code Comments and Identifiers Example
  13. Future Work
  14. Experimental details and resources
  15. Social impact

Figures (1)

  • Figure 1: Comparison of model scores on our adjusted benchmarks and the original CyberSecEval benchmarks. Subplot (a) shows pass percentage, the percentage of samples marked as secure, by the ICD for models originally and after removing prompts that cannot comply with the rules, and (b) reports pass percentage for models originally and after removing comments/identifiers.