Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs

TL;DR

This work addresses the security risk that 3H-aligned LLMs face from poisoned external evidence in retrieval-augmented contexts. It introduces Dialectical Alignment (DA), a framework that enables dialectical reasoning to resolve inter-context and context-memory conflicts by using AI feedback to identify effective reasoning paths, followed by dialectical supervised fine-tuning. DA constructs specialized datasets and prompts, enabling the model to spontaneously decide when to trust external input versus its internal memory, improving poisoned-context defense by about 20 percentage points while preserving in-context knowledge editing capabilities. The approach enhances the robustness and safety of RAG-based LLM deployments, offering actionable mechanisms to mitigate red-team and poisoned-data attacks without requiring defensive prefixes or prompt engineering.

Abstract

With the rise of large language models (LLMs), ensuring they embody the principles of being helpful, honest, and harmless (3H), known as Human Alignment, becomes crucial. While existing alignment methods like RLHF, DPO, etc., effectively fine-tune LLMs to match preferences in the preference dataset, they often lead LLMs to highly receptive human input and external evidence, even when this information is poisoned. This leads to a tendency for LLMs to be Adaptive Chameleons when external evidence conflicts with their parametric memory. This exacerbates the risk of LLM being attacked by external poisoned data, which poses a significant security risk to LLM system applications such as Retrieval-augmented generation (RAG). To address the challenge, we propose a novel framework: Dialectical Alignment (DA), which (1) utilizes AI feedback to identify optimal strategies for LLMs to navigate inter-context conflicts and context-memory conflicts with different external evidence in context window (i.e., different ratios of poisoned factual contexts); (2) constructs the SFT dataset as well as the preference dataset based on the AI feedback and strategies above; (3) uses the above datasets for LLM alignment to defense poisoned context attack while preserving the effectiveness of in-context knowledge editing. Our experiments show that the dialectical alignment model improves poisoned data attack defense by 20 and does not require any additional prompt engineering or prior declaration of ``you may be attacked`` to the LLMs' context window.

Figures (6)

• Figure 1: In-context Knowledge Editting(left) and Poisoned Context Attack (right) are two sides of the coin. An intuitive example illustrates that both knowledge editing based on In-context Learning zheng-etal-2023-edit and attacking a model by injecting posioned data into the context window of LLMs are essentially identical zhong2023poisoningzou2024poisonedrag, differing only in the content of the information in the model's input and the user's purpose.
• Figure 2: Dialectical LLMs. An intuitive example illustrates that both knowledge editing and poisoning data attacks essentially exploit Context-memory conflict. Merely being friendly can lead to LLMs tending to believe the input and change their views (left one), while excessive defensiveness can result in models becoming stubborn because they question external data (right one). However, by learning via our Dialectical Alignment, the model can decide when to update and when to defend (middle one).
• Figure 3: Dialectical Alignment framework. In STEP 1, we use the Base instruction in Table \ref{['tab:gen_path']} to enable the model to answer questions based on the provided context, which may consist of factual or poisoned information (or a combination of both). Once the model provides answers, we use a SOTA LLM as the ground truth to assess the Accuracy (ACC) of the answers (referred to as AI feedback in this figure). In STEP 2, we use other dialectical paths in Table \ref{['tab:gen_path']} to prompt LLMs and provide AI feedback again. Based on this, we select the optimal dialectical path (e.g., CoT-PK, as illustrated in the figure) for different contexts. In STEP 3, we refine these paths using another SOTA LLM if the response is unnatural. Finally, in STEP 4, we construct a supervised fine-tuning dataset using paths corresponding to higher ACC and fine-tune the model.
• Figure 4: PCA Results of TinyDolphin with Different Paths in Section \ref{['sec:da_path']}. FC, RO, and PCN refer to Factual Context and reorder Factual evidence before the poisoned contexts and the number of poisoned contexts, respectively.
• Figure 5: PCA Results of Mistral-7B with Different Paths in Section \ref{['sec:da_path']}.