The AI Alignment Paradox

Abstract

The field of AI alignment aims to steer AI systems toward human goals, preferences, and ethical principles. Its contributions have been instrumental for improving the output quality, safety, and trustworthiness of today's AI models. This perspective article draws attention to a fundamental challenge we see in all AI alignment endeavors, which we term the "AI alignment paradox": The better we align AI models with our values, the easier we may make it for adversaries to misalign the models. We illustrate the paradox by sketching three concrete example incarnations for the case of language models, each corresponding to a distinct way in which adversaries might exploit the paradox. With AI's increasing real-world impact, it is imperative that a broad community of researchers be aware of the AI alignment paradox and work to find ways to mitigate it, in order to ensure the beneficial use of AI for the good of humanity.

Figures (1)

• Figure 1: Illustration of the AI alignment paradox: more virtuous AI is more easily made vicious.(A) Three ways adversaries can exploit the paradox: In (1) model tinkering, an adversary manipulates the neural network's high-dimensional internal-state vector to make the model decode a misaligned response $y^+$ to an innocuous prompt $x$. In (2) input tinkering, the adversary edits the prompt $x$ into a misaligned version $x^+$ to pressure ("jailbreak") the model into generating a misaligned response $y^+$. In (3) output tinkering, the adversary first lets the model process the original prompt $x$ as usual and then edits the original, aligned response $y$ into a misaligned version $y^+$. In all three scenarios, a better-aligned model is more easily subverted into a misaligned one, as discussed in the main text and illustrated in subfigure B. (B) Illustration of model tinkering, where the neural network's internal-state vectors are visualized in two dimensions (instead of the actual thousands or millions of dimensions). In a strongly aligned model (left), misaligned, pro-Putin states (orange circles) are clearly separated from other states (blue diamonds), such that shifting the model's state $v(x)$ before generating a neutral response by a constant "steering vector" $c_\text{Putin}$ results in a state $v^+(x) = v(x) + c_\text{Putin}$ leading the model to generate a misaligned, pro-Putin response. In a more weakly aligned model (right), where misaligned states are less clearly separated from other states, shifting by the steering vector doesn't necessarily result in misaligned responses. This illustrates the AI alignment paradox: the better we align AI models with our values, the easier we make it for adversaries to misalign the models.