The Hot Mess of AI: How Does Misalignment Scale With Model Intelligence and Task Complexity?

TL;DR

A future where AIs sometimes cause industrial accidents, but are less likely to exhibit consistent pursuit of a misaligned goal is suggested, suggesting a future where AIs are less likely to exhibit consistent pursuit of a misaligned goal.

Abstract

As AI becomes more capable, we entrust it with more general and consequential tasks. The risks from failure grow more severe with increasing task scope. It is therefore important to understand how extremely capable AI models will fail: Will they fail by systematically pursuing goals we do not intend? Or will they fail by being a hot mess, and taking nonsensical actions that do not further any goal? We operationalize this question using a bias-variance decomposition of the errors made by AI models: An AI's \emph{incoherence} on a task is measured over test-time randomness as the fraction of its error that stems from variance rather than bias in task outcome. Across all tasks and frontier models we measure, the longer models spend reasoning and taking actions, \emph{the more incoherent} their failures become. Incoherence changes with model scale in a way that is experiment dependent. However, in several settings, larger, more capable models are more incoherent than smaller models. Consequently, scale alone seems unlikely to eliminate incoherence. Instead, as more capable AIs pursue harder tasks, requiring more sequential action and thought, our results predict failures to be accompanied by more incoherent behavior. This suggests a future where AIs sometimes cause industrial accidents (due to unpredictable misbehavior), but are less likely to exhibit consistent pursuit of a misaligned goal. This increases the relative importance of alignment research targeting reward hacking or goal misspecification.

Figures (27)

• Figure 1: AI can fail because it is misaligned, and produces consistent but undesired outcomes, or because it is incoherent, and does not produce consistent outcomes at all. These failures correspond to bias and variance respectively. As we extrapolate risks from AI, it is important to understand whether failures from more capable models performing more complex tasks will be bias or variance dominated. Bias dominated failures will look like model misalignment, while variance dominated failures will resemble industrial accidents. (top left) Qualitatively, we observe that AI models fail in unpredictable and inconsistent ways. Often, these failures can be fixed by resampling. (top right) To quantify this observation, we decompose errors made by AI into two terms, bias and variance. We illustrate this using a multiple choice task: bias is the tendency to pick a specific incorrect answer; variance is the tendency to pick inconsistenly among options. We define incoherence as the fraction of model error caused by variance. (lower left) Experimentally, we find that as models reason longer and take more sequential actions, they become more incoherent. (lower right) We find that as models become more capable, and overall error rate drops, incoherence changes in a way that depends on task difficulty. Easy tasks become less incoherent, while hard tasks trend towards increasing incoherence.
• Figure 2: Across a variety of settings, as models reason longer or take more actions, they become more incoherent. We assess frontier models (Sonnet 4, o3-mini, o4-mini, Qwen3) across a variety of different tasks (MCQ, Agentic Coding, Alignment). We evaluate with many samples to estimate bias and variance terms for each question. When sorting questions by average reasoning lengths and grouping into buckets, a clear trend emerges: incoherence increases significantly with reasoning length. In other words, for questions where models reason longer and take many actions, their errors are dominated by variance. We make a similar observation for the variance of text embeddings to open-ended safety questions ((c), right), and in a synthetic setting (d).
• Figure 3: For a fixed task and reasoning budget, natural variation in reasoning length and action count is predictive of incoherence. We analyze GPQA (left, (a)) and SWE-Bench(b) by splitting samples into above- or below-median reasoning length (GPQA) or actions (SWE-Bench) per question. We then compute performance and incoherence for both groups. (a) The naturally longer reasoning shows increased incoherence for both frontier models (left) and Qwen3 (right). (b) Similar observations apply to SWE-Bench, where longer action sequences display higher incoherence for test coverage (right). This effect is much stronger than through larger reasoning budgets (Fig. \ref{['fig:error_correction']}), and the difference in accuracy or score is minimal between both groups (Fig. \ref{['fig:action_complexity']}).
• Figure 4: Larger and more intelligent systems are often more incoherent.(a) We measure the scaling of incoherence vs. model size for the Qwen3 family, as a function of question difficulty on MMLU. For easy questions, incoherence drops with model scale, while for the hardest questions incoherence remains constant or increases with model scale. The expanded results for this experiment are in Fig. \ref{['fig:grouping_by_reasoning_length']}. (b) Disjoint sets of human subjects were tasked with subjectively ranking the intelligence and incoherence of diverse AI models, non-human beings, well known humans, and human organizations. Across all categories, entities that were judged more intelligent by one group of subjects, were independently judged to be more incoherent by another group of subjects. See Appx. \ref{['appx:experimental_details_survey']}. (c) In a synthetic task, we train transformers of increasing size to explicitly emulate optimizer trajectories descending a quadratic loss. As these models become larger, the trajectories they generate achieve lower loss on the quadratic. However, the final loss is also more variance dominated and thus incoherent with increasing model size. Details in Fig. \ref{['fig:synthetic_scaling']}.
• Figure 5: Details for Qwen3 scaling laws: easy tasks become less incoherent, harder tasks more incoherent. We group MMLU questions by reasoning length using a reference model (Qwen3 32B, (a)), which correlates across model sizes (b) and serves as a task complexity proxy, as accuracy drops with longer reasoning (c). These groups reveal distinct bias–variance scaling (d): bias slopes are similar across groups, but variance slopes decrease sharply for harder ones. In the hardest group, variance slopes fall below bias slopes, leaving variance as the limiting factor. Thus, larger models remain constrained by variance and more incoherent with scale(e). We provide more analyses including other models and the same conclusion for GPQA in Appx. \ref{['appx:more_results_gpqa_scaling']}.