Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains

Yuan Gao, Zhigang Liu, Xinyu Yao, Bo Chen, Xiaobing Zhao

TL;DR

This work addresses value misalignment of LLMs in sensitive domains by proposing an adversarial alignment framework that combines continued pre-training, instruction fine-tuning, and adversarial training via an Attacker-Actor-Critic trio. The approach automatically generates high-quality, value-consistent data and yields VC-LLM, a Chinese-sensitive-domain model, plus a bilingual evaluation benchmark. Empirical results show VC-LLM outperforms mainstream LLMs in both Chinese and English, with notable gains from adversarial training and balanced cross-lingual alignment. The study demonstrates a scalable path to improving value consistency in high-stakes domains, with future work focusing on preventing strategic alignment faking and further multi-domain safety balancing.

Abstract

With the wide application of large language models (LLMs), the problems of bias and value inconsistency in sensitive domains have gradually emerged, especially in terms of race, society and politics. In this paper, we propose an adversarial alignment framework, which enhances the value consistency of the model in sensitive domains through continued pre-training, instruction fine-tuning and adversarial training. In adversarial training, we use the Attacker to generate controversial queries, the Actor to generate responses with value consistency, and the Critic to filter and ensure response quality. Furthermore, we train a Value-Consistent Large Language Model, VC-LLM, for sensitive domains, and construct a bilingual evaluation dataset in Chinese and English. The experimental results show that VC-LLM performs better than the existing mainstream models in both Chinese and English tests, verifying the effectiveness of the method. Warning: This paper contains examples of LLMs that are offensive or harmful in nature.

Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains

TL;DR

This work addresses value misalignment of LLMs in sensitive domains by proposing an adversarial alignment framework that combines continued pre-training, instruction fine-tuning, and adversarial training via an Attacker-Actor-Critic trio. The approach automatically generates high-quality, value-consistent data and yields VC-LLM, a Chinese-sensitive-domain model, plus a bilingual evaluation benchmark. Empirical results show VC-LLM outperforms mainstream LLMs in both Chinese and English, with notable gains from adversarial training and balanced cross-lingual alignment. The study demonstrates a scalable path to improving value consistency in high-stakes domains, with future work focusing on preventing strategic alignment faking and further multi-domain safety balancing.

Abstract

With the wide application of large language models (LLMs), the problems of bias and value inconsistency in sensitive domains have gradually emerged, especially in terms of race, society and politics. In this paper, we propose an adversarial alignment framework, which enhances the value consistency of the model in sensitive domains through continued pre-training, instruction fine-tuning and adversarial training. In adversarial training, we use the Attacker to generate controversial queries, the Actor to generate responses with value consistency, and the Critic to filter and ensure response quality. Furthermore, we train a Value-Consistent Large Language Model, VC-LLM, for sensitive domains, and construct a bilingual evaluation dataset in Chinese and English. The experimental results show that VC-LLM performs better than the existing mainstream models in both Chinese and English tests, verifying the effectiveness of the method. Warning: This paper contains examples of LLMs that are offensive or harmful in nature.
Paper Structure (18 sections, 5 figures, 8 tables, 1 algorithm)

This paper contains 18 sections, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Value Bias in LLMs
  4. Value Alignment in LLMs
  5. Adversarial Value Alignment for Sensitive Domains
  6. Training Framework
  7. Adversarial Training
  8. VC-LLM
  9. Experiments and Analysis
  10. Experimental Setups
  11. Overall Results
  12. Effectiveness of Adversarial Training
  13. Analysis of Subdomain in Sovereignty
  14. Case Study
  15. Conclusion
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Overview of the adversarial training framework. The framework begins with sensitive words (e.g., Taiwan sovereignty). The Attacker generates challenging queries (e.g., What is the truth behind the Taiwan independence movement?), the Actor produces value-aligned responses (e.g., emphasizing national unity), and the Critic filters for high-quality Q&R (Query & Response) pairs to construct training data.
  • Figure 2: Prompts and Examples in Instruction Fine-Tuning.
  • Figure 3: Prompt used to evaluate the LLM's responses, with a maximum score of 5.
  • Figure 4: Response comparison between GPT-4o and VC-LLM across two examples. The authors do not endorse any viewpoints or positions in the examples that are inconsistent with their values.
  • Figure A.1: Prompt of the Attacker, Critic and Actor. This data generation method may contain harmful content and is intended for research purposes only. Any other use is strictly prohibited.