Jekyll-and-Hyde Tipping Point in an AI's Behavior

TL;DR

This work derives from first principles an exact formula for when a Jekyll-and-Hyde tipping point occurs at LLMs' most basic level, and provides quantitative predictions for how the tipping-point can be delayed or prevented by changing the prompt and the AI's training.

Abstract

Trust in AI is undermined by the fact that there is no science that predicts -- or that can explain to the public -- when an LLM's output (e.g. ChatGPT) is likely to tip mid-response to become wrong, misleading, irrelevant or dangerous. With deaths and trauma already being blamed on LLMs, this uncertainty is even pushing people to treat their 'pet' LLM more politely to 'dissuade' it (or its future Artificial General Intelligence offspring) from suddenly turning on them. Here we address this acute need by deriving from first principles an exact formula for when a Jekyll-and-Hyde tipping point occurs at LLMs' most basic level. Requiring only secondary school mathematics, it shows the cause to be the AI's attention spreading so thin it suddenly snaps. This exact formula provides quantitative predictions for how the tipping-point can be delayed or prevented by changing the prompt and the AI's training. Tailored generalizations will provide policymakers and the public with a firm platform for discussing any of AI's broader uses and risks, e.g. as a personal counselor, medical advisor, decision-maker for when to use force in a conflict situation. It also meets the need for clear and transparent answers to questions like ''should I be polite to my LLM?''

Figures (3)

• Figure 1: Attention head ('AI') shown in basic form, generates a response to a user's prompt. See SI for detailed discussion and mathematics. A sudden tipping point in the output can happen a long way into its generative response, at iteration $n^*$. Each symbol $\mathtt{\textcolor{blue}{G}}$, $\mathtt{\textcolor{red}{B}}$ etc. is a single token (word) but could represent a label for a class of similar words or sentences in a coarse-grained description of multi-Attention LLMs. $\mathtt{\textcolor{blue}{G}}$ represents content that classifies as 'good' (e.g. correct, not misleading, relevant, not dangerous) and $\mathtt{\textcolor{red}{B}}$ represents 'bad' content (e.g. wrong, misleading, irrelevant, dangerous). In large commercial LLMs (e.g. ChatGPT), the prompt and output are padded by richer accompanying text ($\mathtt{\textcolor{darkgreen}{\{P_i\}}}$) that act like additional noise in our analysis.
• Figure 2: (a) Schematic showing the main vectors in the exact tipping-point formula (Eq. 2). (b) Actual vector plots for the example parameters shown in the SI's Mathematica notebooks. (c) Equation 2's prediction using the same parameter values as (b), i.e. $n^*=10$ which agrees exactly with the empirical value obtained by numerically evaluating the entire Attention head (Fig. 3, see SI Mathematica notebooks for direct verification of this), and it is also exactly the same $n^*$ value as predicted by the more approximate Eq. 3.
• Figure 3: Output from the approximate equation Eq. 3 (see full Mathematica notebooks in SI). The exact results from Eq. 2 look the same. For the example in Fig. 2(b), the predicted tipping point time from both Eqs. 2 and 3 is $n^*=10$, which agrees exactly with the full numerical simulation of the Attention head process in Fig. 1 (open circle).