RACA: Representation-Aware Coverage Criteria for LLM Safety Testing

TL;DR

Safety testing for LLMs is hampered by static jailbreak datasets and the scalability limits of neuron-level coverage criteria. RACA introduces representation-aware coverage, using a calibration set to identify safety-critical representations and PCA to derive Safety Concepts, with concept activations f_j(x) = v_j^T (h(x) - μ) for top components. Coverage is computed via six sub-criteria across two dimensions (Dimension I: individual concepts; Dimension II: compositional concepts), enabling scalable, jailbreak-sensitive evaluation and practical applications like test prioritization and attack prompt sampling. Experiments across multiple models and datasets show that RACA identifies high-quality jailbreak prompts and outperforms neuron-based baselines while generalizing to larger models and varying calibration-set sizes, making it a robust framework for real-world LLM safety testing.

Abstract

Recent advancements in LLMs have led to significant breakthroughs in various AI applications. However, their sophisticated capabilities also introduce severe safety concerns, particularly the generation of harmful content through jailbreak attacks. Current safety testing for LLMs often relies on static datasets and lacks systematic criteria to evaluate the quality and adequacy of these tests. While coverage criteria have been effective for smaller neural networks, they are not directly applicable to LLMs due to scalability issues and differing objectives. To address these challenges, this paper introduces RACA, a novel set of coverage criteria specifically designed for LLM safety testing. RACA leverages representation engineering to focus on safety-critical concepts within LLMs, thereby reducing dimensionality and filtering out irrelevant information. The framework operates in three stages: first, it identifies safety-critical representations using a small, expert-curated calibration set of jailbreak prompts. Second, it calculates conceptual activation scores for a given test suite based on these representations. Finally, it computes coverage results using six sub-criteria that assess both individual and compositional safety concepts. We conduct comprehensive experiments to validate RACA's effectiveness, applicability, and generalization, where the results demonstrate that RACA successfully identifies high-quality jailbreak prompts and is superior to traditional neuron-level criteria. We also showcase its practical application in real-world scenarios, such as test set prioritization and attack prompt sampling. Furthermore, our findings confirm RACA's generalization to various scenarios and its robustness across various configurations. Overall, RACA provides a new framework for evaluating the safety of LLMs, contributing a valuable technique to the field of testing for AI.

Figures (1)

• Figure 1: An overview of our RACA framework. In the Safety Concept Extraction module, we use a Calibration Set to extract safety-critical representations, and then apply PCA to obtain the Safety Principal Components, which serve as the Safety Concepts for subsequent steps. Then, in the Coverage Criteria module, we extract the representation vectors of the Test Suite and compute the final coverage scores with safety concept activations. These scores are derived from six distinct criteria categorized into two dimensions: Individual and Compositional Concept Coverage, aligning with our three core design principles. In practical applications, RACA can be leveraged for Test Prioritization and Attack Prompt Sampling.

Theorems & Definitions (1)

• Definition 1: Large Language Model, LLM