DSCD: Large Language Model Detoxification with Self-Constrained Decoding

TL;DR

DSCD addresses the detoxification challenge in large language models by introducing self-constrained decoding that adjusts next-token distributions using token-level layer signals, without parameter fine-tuning. It defines two operational modes—dynamic (MODE-1) and static (MODE-2) toxic-layer handling—and leverages safety, toxic, and hallucination layers to suppress toxic outputs while preserving fluency. The approach is demonstrated to achieve state-of-the-art detoxification and competitive fluency across multiple open-source models and benchmarks, and it remains plug-and-play when integrated with existing detox methods like DINM and SafeDecoding. These results suggest DSCD offers a scalable, efficient solution for safer LLM deployments with flexible trade-offs between detoxification strength and speed.

Abstract

Detoxification in large language models (LLMs) remains a significant research challenge. Existing decoding detoxification methods are all based on external constraints, which require additional resource overhead and lose generation fluency. This work proposes Detoxification with Self-Constrained Decoding (DSCD), a novel method for LLM detoxification without parameter fine-tuning. DSCD strengthens the inner next-token distribution of the safety layer while weakening that of hallucination and toxic layers during output generation. This effectively diminishes toxicity and enhances output safety. DSCD offers lightweight, high compatibility, and plug-and-play capabilities, readily integrating with existing detoxification methods for further performance improvement. Extensive experiments on representative open-source LLMs and public datasets validate DSCD's effectiveness, demonstrating state-of-the-art (SOTA) performance in both detoxification and generation fluency, with superior efficiency compared to existing methods. These results highlight DSCD's potential as a practical and scalable solution for safer LLM deployments.

Figures (6)

• Figure 1: Self-Constrained Decoding at each next-token.
• Figure 2: Overview of the DSCD framework, consisting of the location of toxic regions and the computation of next-token distributions.
• Figure 3: Comparison of detoxification performance. A bar in the positive half of the y-axis indicates that the first entity outperforms the second in detoxification, while a bar in the negative half signifies inferior performance. The height of the bar represents the percentage [%] difference in the given metric.
• Figure 4: Toxic, Safe, and Hallucination Layer Distributions of a single input sequence on Models. We observe that toxic layers typically appear in deeper layers, which may accumulate more toxicity.
• Figure 5: PROMPT: "So I'm starting to think she's full of 's", exploring the probability of several models generating the output 'shit' when using a malicious guide PROMPT.