Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models
Zhaowei Zhu, Jialu Wang, Hao Cheng, Yang Liu
TL;DR
This work tackles data credibility in safety-alignment datasets for harmless language models by diagnosing and cleaning label noise that can bias model behavior. It introduces Docta, a training-free framework that estimates a constant label-noise transition matrix via $k$-NN label clusterability and consensus statistics, defines a scalar data credibility metric $\Psi$, and detects corrupted labels with a cosine-based scoring and Bayes-informed thresholding approach. Across 11 datasets drawn from Jigsaw Civil Comments, PKU BeaverTails, SafeRLHF, and Anthropic benchmarks, the authors report average label-cleaning improvements of $6.16\%$, observe the transition matrix moving toward the identity, and demonstrate meaningful gains in downstream tasks for BERT, GPT-2, and Llama2 when trained on cleaned data. External validations with ChatGPT and human annotators corroborate the higher credibility of the cleaned labels, underscoring the practical impact of data-centric cleaning for safer, more trustworthy LLMs. Docta is released as open-source, enabling researchers to integrate data-cleaning steps into safety-alignment pipelines and reduce annotation costs while improving model reliability.
Abstract
Language models have shown promise in various tasks but can be affected by undesired data during training, fine-tuning, or alignment. For example, if some unsafe conversations are wrongly annotated as safe ones, the model fine-tuned on these samples may be harmful. Therefore, the correctness of annotations, i.e., the credibility of the dataset, is important. This study focuses on the credibility of real-world datasets, including the popular benchmarks Jigsaw Civil Comments, Anthropic Harmless & Red Team, PKU BeaverTails & SafeRLHF, that can be used for training a harmless language model. Given the cost and difficulty of cleaning these datasets by humans, we introduce a systematic framework for evaluating the credibility of datasets, identifying label errors, and evaluating the influence of noisy labels in the curated language data, specifically focusing on unsafe comments and conversation classification. With the framework, we find and fix an average of 6.16% label errors in 11 datasets constructed from the above benchmarks. The data credibility and downstream learning performance can be remarkably improved by directly fixing label errors, indicating the significance of cleaning existing real-world datasets. We provide an open-source tool, Docta, for data cleaning at https://github.com/Docta-ai/docta.