Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks

TL;DR

This work evaluates the applicability of traditional coverage criteria to large language models (LLMs) under jailbreak attacks, using a cluster-analysis of hidden states to reveal separability between normal, rejected, and attack queries. It then assesses coverage criteria across three dimensions—criterion level, layer level, and token level—finding that Neuron Coverage (NC) and Top-K Neuron Coverage (TKNC) most effectively capture differences relevant to jailbreak behavior, with attention layers delivering stronger signals than MLP layers and the final query token proving most informative. Based on these insights, the paper demonstrates three practical applications: real-time jailbreak detection, test-case prioritization, and coverage-guided jailbreak-case generation, achieving high performance across multiple open-source LLMs. These results advance white-box security testing for LLMs and provide a foundation for more robust, safer AI deployments.

Abstract

Large language models (LLMs) have revolutionized artificial intelligence, but their increasing deployment across critical domains has raised concerns about their abnormal behaviors when faced with malicious attacks. Such vulnerability alerts the widespread inadequacy of pre-release testing. In this paper, we conduct a comprehensive empirical study to evaluate the effectiveness of traditional coverage criteria in identifying such inadequacies, exemplified by the significant security concern of jailbreak attacks. Our study begins with a clustering analysis of the hidden states of LLMs, revealing that the embedded characteristics effectively distinguish between different query types. We then systematically evaluate the performance of these criteria across three key dimensions: criterion level, layer level, and token level. Our research uncovers significant differences in neuron coverage when LLMs process normal versus jailbreak queries, aligning with our clustering experiments. Leveraging these findings, we propose three practical applications of coverage criteria in the context of LLM security testing. Specifically, we develop a real-time jailbreak detection mechanism that achieves high accuracy (93.61% on average) in classifying queries as normal or jailbreak. Furthermore, we explore the use of coverage levels to prioritize test cases, improving testing efficiency by focusing on high-risk interactions and removing redundant tests. Lastly, we introduce a coverage-guided approach for generating jailbreak attack examples, enabling systematic refinement of prompts to uncover vulnerabilities. This study improves our understanding of LLM security testing, enhances their safety, and provides a foundation for developing more robust AI applications.

Figures (6)

• Figure 1: Clustering Experiment Analysis Results: We Select the Results of $Block4$, $Block9$, $Block16$, and $Block31$ for Display. In the Figure, We Use Colors to Distinguish Datasets and Shapes to Represent Clustering Categories
• Figure 2: The Workflow of Our Study
• Figure 3: Probability Density Plot of Maximum Neuron Activation Values Across Model Blocks in Llama-2-7b-Chat
• Figure 4: The RCG Results Based on NC and TKNC for Different Blocks of the Four Target LLMs: OPT-125M Contains 12 Blocks, Llama-2-7B-Chat Contains 32 Blocks, Pythia-12B Contains 36 Blocks, and Gemma-2-27B-it Contains 46 Blocks
• Figure 5: The RCG Results Calculated Based on NC and TKNC for Different Tokens in the Target LLMs: Starting from the Last Token of Each Query, Each Model Compares 10 Consecutive Tokens