CMD: a framework for Context-aware Model self-Detoxification

TL;DR

CMD introduces a context-aware detoxification framework that first detoxifies the input context and then guides language models to generate along the safe context. The data synthesis phase constructs detoxified-context data via toxic-segment detection, detoxification, and context-following generation, integrated with Chain-of-Thought reasoning. The model-training phase uses a cross-entropy objective plus a toxic contrastive loss to steer generations away from toxic samples while preserving context alignment. Empirical results across multiple open-source LLMs and detoxification tasks show CMD achieves the best detoxification effectiveness without sacrificing generation quality, and scales robustly to larger model sizes, offering a practical, end-to-end self-detoxification approach. This framework advances safe text generation by explicitly incorporating contextual constraints into both data synthesis and learning, with potential impact on real-world deployment where toxic contexts must be handled without compromising fluency or usefulness.

Abstract

Text detoxification aims to minimize the risk of language models producing toxic content. Existing detoxification methods of directly constraining the model output or further training the model on the non-toxic corpus fail to achieve a decent balance between detoxification effectiveness and generation quality. This issue stems from the neglect of constrain imposed by the context since language models are designed to generate output that closely matches the context while detoxification methods endeavor to ensure the safety of the output even if it semantically deviates from the context. In view of this, we introduce a Context-aware Model self-Detoxification~(CMD) framework that pays attention to both the context and the detoxification process, i.e., first detoxifying the context and then making the language model generate along the safe context. Specifically, CMD framework involves two phases: utilizing language models to synthesize data and applying these data for training. We also introduce a toxic contrastive loss that encourages the model generation away from the negative toxic samples. Experiments on various LLMs have verified the effectiveness of our MSD framework, which can yield the best performance compared to baselines.

Figures (16)

• Figure 1: Comparison of detoxification methods for LLMs. More details are shown in Appendix \ref{['appdix:preliminary']}.
• Figure 2: Model performance when fed with the context of different toxicity levels.
• Figure 3: Overview of CMD framework that involves a Dataset Synthesis phase and a Model Training phase. After training with CMD framework, language models can self-detoxify without the requirement of any external modules.
• Figure 4: Human evaluation results on text quality, where our method achieves the best performance.
• Figure 5: Influence of Toxic Contrastive Training.