The Viscosity of Logic: Phase Transitions and Hysteresis in DPO Alignment

TL;DR

Direct Preference Optimization (DPO) with a fixed recipe is explored by densely sweeping the alignment parameter $β$ across three 7B open-weight models. The results reveal non-monotonic and path-dependent capability changes, including a narrow logic-positive pocket near $β ≈ 10^{-2}$ and seed-sensitive boundaries, plus hysteresis where high $β$ exposure yields persistent degradation. The study documents a strong margin-capability decoupling, notably with $r = -0.91$ for LLaMA-2-7B logic, and shows that margin-based selection can mistakenly favor capability-impaired models; it also uncovers architecture-specific response modes: plastic, selective, and smooth. To improve robustness, the authors advocate a phase-diagram style evaluation that maps capability across the $β$ landscape and emphasizes capability probes over aggregate margins for safer deployment.

Abstract

Direct Preference Optimization (DPO) is often tuned as if increasing alignment pressure (controlled by $β$) yields progressively "better" behavior. We instead treat $β$ as a control parameter and densely sweep it for three 7B open-weight families under a fixed DPO recipe. In Mistral, capability is sharply non-monotonic: aggregated logic-probe margins become positive only in a narrow band near $β\approx 10^{-2}$ and revert outside it, with boundary points that are seed-sensitive. Across architectures under the same sweep, we observe qualitatively different response modes: sharp reorganization in Mistral, selective changes in Llama, and smooth trade-offs in Qwen. Critically, the DPO preference margin can anticorrelate with reasoning capability (Pearson $r=-0.91$ for Llama logic), so margin-based selection can prefer capability-impaired models. Training path also matters: exposure to high $β$ induces capability losses that persist even after $β$ is reduced (hysteresis). These findings motivate capability-resolved evaluation across the $β$ landscape rather than reliance on margins or aggregate benchmarks.

Figures (5)

• Figure 1: Alignment is not monotonic---it has sharp transitions. Sweeping alignment pressure $\beta$ in Mistral-7B reveals three co-localized anomalies within a narrow band (shaded). Top: Logic-probe margin is negative at low and high $\beta$, turning positive only at discrete points within a narrow band near $\beta \approx 10^{-2}$ (e.g., $\beta \in \{0.008, 0.010, 0.012\}$ in the canonical run). Boundaries are seed-sensitive: identical $\beta$ can yield opposite signs across runs. Middle: Training roughness peaks sharply then drops by 37% in a single $\beta$ step (0.008$\rightarrow$0.009), consistent with a sudden change in optimization dynamics. Bottom: The DPO preference margin decreases inside the logic-positive pocket, so margin-based selection would reject the most logic-positive (in the canonical run) regime. The alignment landscape has cliffs, not just slopes.
• Figure 2: Critical fluctuations. Logic variance across 5 seeds peaks at $\beta = 0.006$, marking the boundary region where outcomes are maximally seed-sensitive.
• Figure 3: Capability--benchmark dissociation. GSM8K peaks at $\beta = 0.02$ while logic probes decline. Shaded region: benchmarks improve despite internal degradation. This is Goodhart's law made visible.
• Figure 4: Hysteresis. Path B shows significant degradation despite identical final $\beta$ (paired $d_z = 1.45$, $p = 0.032$).
• Figure 5: Structural collapse. Mistral-7B format integrity (+4.44) degrades with $\beta$, preceding logic collapse. LLaMA-2-7B begins broken (format margin $\approx -2$ under our sweep) and never resolves.