Paying Alignment Tax with Contrastive Learning

TL;DR

The paper addresses the alignment tax problem in debiasing large language models, where toxicity reduction often degrades factual accuracy and knowledge retention. It introduces a contrastive learning framework that explicitly models both positive (unbiased) and negative (biased) outputs using targeted data augmentation and a dynamic loss formulation to balance debiasing with faithfulness. Key contributions include a comprehensive benchmark for measuring alignment tax, an open-source contrastive debiasing method, and empirical evidence showing simultaneous improvements in toxicity reduction and faithfulness across model scales, with benefits scaling to larger architectures. The work suggests that explicit boundary modeling via contrastive learning could reduce alignment tax and generalize to broader AI alignment challenges beyond bias mitigation.

Abstract

Current debiasing approaches often result a degradation in model capabilities such as factual accuracy and knowledge retention. Through systematic evaluation across multiple benchmarks, we demonstrate that existing debiasing methods face fundamental trade-offs, particularly in smaller models, leading to reduced truthfulness, knowledge loss, or unintelligible outputs. To address these limitations, we propose a contrastive learning framework that learns through carefully constructed positive and negative examples. Our approach introduces contrast computation and dynamic loss scaling to balance bias mitigation with faithfulness preservation. Experimental results across multiple model scales demonstrate that our method achieves substantial improvements in both toxicity reduction and faithfulness preservation. Most importantly, we show that our framework is the first to consistently improve both metrics simultaneously, avoiding the capability degradation characteristic of existing approaches. These results suggest that explicit modeling of both positive and negative examples through contrastive learning could be a promising direction for reducing the alignment tax in language model debiasing.