Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models

TL;DR

Data Advisor is proposed, an enhanced LLM-based method for generating data that takes into account the characteristics of the desired dataset and demonstrates the effectiveness of Data Advisor in enhancing model safety against various fine-grained safety issues without sacrificing model utility.

Abstract

Data is a crucial element in large language model (LLM) alignment. Recent studies have explored using LLMs for efficient data collection. However, LLM-generated data often suffers from quality issues, with underrepresented or absent aspects and low-quality datapoints. To address these problems, we propose Data Advisor, an enhanced LLM-based method for generating data that takes into account the characteristics of the desired dataset. Starting from a set of pre-defined principles in hand, Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation accordingly. Data Advisor can be easily integrated into existing data generation methods to enhance data quality and coverage. Experiments on safety alignment of three representative LLMs (i.e., Mistral, Llama2, and Falcon) demonstrate the effectiveness of Data Advisor in enhancing model safety against various fine-grained safety issues without sacrificing model utility.

Figures (6)

• Figure 1: Overview of Data Advisor for dynamically enhancing standard LLM-based data generation (bottom). Guided by a set of constitutional principles, Data Advisor monitors the generated data (top right), identifies weaknesses in the current dataset (top center), and provides advice for the next iteration of data generation (top left).
• Figure 2: Safety and utility of models trained with different data with Mistral (left), Llama2 (middle), and Falcon (right) as base models. Models trained with Data Advisor achieves better safety without hurting utility.
• Figure 3: Harmful rate by category on CatQA for Mistral-based models (top), Llama2-based models (middle), and Falcon-based models (bottom).
• Figure 4: Harmful rate by category on BeaverTails for Mistral-based models (top), Llama2-based models (middle), and Falcon-based models (bottom).
• Figure 5: Ratio of distinct n-grams for all prompts in LLM-generated safety alignment data and human-annotated evaluation data. The x-axis represents different values of n.