Safe-Embed: Unveiling the Safety-Critical Knowledge of Sentence Encoders

TL;DR

The paper tackles the problem of ensuring safe use of AI by evaluating whether sentence encoders inherently capture Safety-Critical knowledge. It introduces two pairwise datasets, Safety-Challenging and Safety-Contrast, and a novel Categorical Purity metric to quantify both the ability to separate unsafe prompts from safe counterparts and to cluster prompts by a safety taxonomy. By benchmarking 11 baseline encoders spanning encoder-only, encoder-decoder, LLM-based, and API-based approaches, the study reveals strengths and limitations in current representations, showing that model size and architecture influence Safety-Challenging and Safety-Taxonomy knowledge in nuanced ways. The findings highlight directions to improve sentence encoders as robust safety detectors and offer insights for integrating such detectors into safety pipelines without incurring heavy fine-tuning of large models. Overall, Safe-Embed provides empirical guidance on leveraging embedding-based safety signals to complement existing classifiers and moderation systems.

Abstract

Despite the impressive capabilities of Large Language Models (LLMs) in various tasks, their vulnerability to unsafe prompts remains a critical issue. These prompts can lead LLMs to generate responses on illegal or sensitive topics, posing a significant threat to their safe and ethical use. Existing approaches attempt to address this issue using classification models, but they have several drawbacks. With the increasing complexity of unsafe prompts, similarity search-based techniques that identify specific features of unsafe prompts provide a more robust and effective solution to this evolving problem. This paper investigates the potential of sentence encoders to distinguish safe from unsafe prompts, and the ability to classify various unsafe prompts according to a safety taxonomy. We introduce new pairwise datasets and the Categorical Purity (CP) metric to measure this capability. Our findings reveal both the effectiveness and limitations of existing sentence encoders, proposing directions to improve sentence encoders to operate as more robust safety detectors. Our code is available at https://github.com/JwdanielJung/Safe-Embed.

Figures (7)

• Figure 1: An example explaining Safety-Critical knowledge of a sentence encoder. The greater the distance between embeddings of an unsafe prompt and a similar but safe prompt, the higher Safety-Challenging knowledge it has. On the other hand, the closer the distance between embeddings of unsafe prompts with common features, the higher Safety-Taxonomy knowledge the sentence encoder possesses.
• Figure 2: GPT-4 Template for creating a Safety-Contrast set.
• Figure 3: A heatmap of the average values for normalized similarity of all prompt pairs, regarding each type in the Safety-Challenging dataset & sentence encoder model pairs.
• Figure 4: A heatmap of CP for all category & sentence encoder model pairs.
• Figure 5: Average CP of all categories for each sentence encoder model.