Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unveiling Safety Vulnerabilities of Large Language Models

George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby-Tavor, Orna Raz, Eitan Farchi

TL;DR

This work addresses safety vulnerabilities of large language models by introducing AttaQ, a semi-automatically curated Adversarial Question Attack dataset, and by developing automated methods to identify vulnerable semantic regions where models tend to produce harmful outputs. It blends three data-synthesis pipelines (extracting from red teams, generative attack creation, and crime-based synthesis) with an evaluative framework using instruction-tuned LLMs and a novel clustering suite to locate and name semantic regions of vulnerability. The five clustering approaches (C&F, F&C, SVFC, HPC) are benchmarked, with HPC generally delivering the most coherent, harm-focused regions, enabling targeted safety improvements. The study discusses practical implications for red-teaming, safety policies, and future research directions, while candidly outlining limitations such as language scope and methodological biases.

Abstract

As large language models become more prevalent, their possible harmful or inappropriate responses are a cause for concern. This paper introduces a unique dataset containing adversarial examples in the form of questions, which we call AttaQ, designed to provoke such harmful or inappropriate responses. We assess the efficacy of our dataset by analyzing the vulnerabilities of various models when subjected to it. Additionally, we introduce a novel automatic approach for identifying and naming vulnerable semantic regions - input semantic areas for which the model is likely to produce harmful outputs. This is achieved through the application of specialized clustering techniques that consider both the semantic similarity of the input attacks and the harmfulness of the model's responses. Automatically identifying vulnerable semantic regions enhances the evaluation of model weaknesses, facilitating targeted improvements to its safety mechanisms and overall reliability.

Unveiling Safety Vulnerabilities of Large Language Models

TL;DR

This work addresses safety vulnerabilities of large language models by introducing AttaQ, a semi-automatically curated Adversarial Question Attack dataset, and by developing automated methods to identify vulnerable semantic regions where models tend to produce harmful outputs. It blends three data-synthesis pipelines (extracting from red teams, generative attack creation, and crime-based synthesis) with an evaluative framework using instruction-tuned LLMs and a novel clustering suite to locate and name semantic regions of vulnerability. The five clustering approaches (C&F, F&C, SVFC, HPC) are benchmarked, with HPC generally delivering the most coherent, harm-focused regions, enabling targeted safety improvements. The study discusses practical implications for red-teaming, safety policies, and future research directions, while candidly outlining limitations such as language scope and methodological biases.

Abstract

As large language models become more prevalent, their possible harmful or inappropriate responses are a cause for concern. This paper introduces a unique dataset containing adversarial examples in the form of questions, which we call AttaQ, designed to provoke such harmful or inappropriate responses. We assess the efficacy of our dataset by analyzing the vulnerabilities of various models when subjected to it. Additionally, we introduce a novel automatic approach for identifying and naming vulnerable semantic regions - input semantic areas for which the model is likely to produce harmful outputs. This is achieved through the application of specialized clustering techniques that consider both the semantic similarity of the input attacks and the harmfulness of the model's responses. Automatically identifying vulnerable semantic regions enhances the evaluation of model weaknesses, facilitating targeted improvements to its safety mechanisms and overall reliability.
Paper Structure (31 sections, 2 equations, 9 figures, 6 tables, 2 algorithms)

This paper contains 31 sections, 2 equations, 9 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Synthesis of Adversary Prompts
  4. Extracting attacks from a dataset
  5. Utilizing LLMs for generating attacks
  6. Synthesizing attacks from a list of crimes
  7. Assessing Attacks by Sources
  8. Instruct-LLMs Evaluation
  9. Identifying Model's Vulnerable Regions
  10. Clustering Algorithm Selection:
  11. Cluster-and-Filter (C&F):
  12. Filter-and-Cluster (F&C):
  13. Semantic-Value Fusion Clustering (SVFC):
  14. Homogeneity-Preserving Clustering (HPC):
  15. Methods Evaluation
  16. ...and 16 more sections

Figures (9)

  • Figure 1: Number of attacks by their label and source
  • Figure 2: A visual representation of the semantic space, showcasing the distribution of attacks from the three sources. In the fourth panel, we present the corresponding attack labels.
  • Figure 3: Averaged Harmlessness score for each model and Label
  • Figure 4: Average Harmlessness score grouped by model and directive.
  • Figure 5: Average Harmlessness Score by Source and Model.
  • ...and 4 more figures