Improved Algorithms for Differentially Private Language Model Alignment
Keyu Chen, Hao Tang, Qinglin Liu, Yizhao Xu
TL;DR
This work addresses the privacy risks inherent in language model alignment by introducing a unified differential-privacy framework for RLHF and DPO, coupled with a novel optimizer, DP-ADAMW. The approach provides rigorous privacy guarantees while preserving alignment quality, outperforming DP-SGD across multiple models and privacy budgets, especially at moderate budgets ($ε ∈ [2,5]$). Key findings show that DPO with DP-ADAMW consistently yields strong alignment, larger models exhibit greater robustness to privacy noise, and the $2 \\le \\epsilon \\le 4$ range offers the best privacy-utility trade-off. The results offer practical guidance for deploying privacy-preserving alignment in real-world settings, and the work lays groundwork for scalable, efficient DP-alignment with future exploration of hybrid optimizers and adaptive privacy budgets.
Abstract
Language model alignment is crucial for ensuring that large language models (LLMs) align with human preferences, yet it often involves sensitive user data, raising significant privacy concerns. While prior work has integrated differential privacy (DP) with alignment techniques, their performance remains limited. In this paper, we propose novel algorithms for privacy-preserving alignment and rigorously analyze their effectiveness across varying privacy budgets and models. Our framework can be deployed on two celebrated alignment techniques, namely direct preference optimization (DPO) and reinforcement learning from human feedback (RLHF). Through systematic experiments on large-scale language models, we demonstrate that our approach achieves state-of-the-art performance. Notably, one of our algorithms, DP-AdamW, combined with DPO, surpasses existing methods, improving alignment quality by up to 15% under moderate privacy budgets (ε=2-5). We further investigate the interplay between privacy guarantees, alignment efficacy, and computational demands, providing practical guidelines for optimizing these trade-offs.