Steerability of Instrumental-Convergence Tendencies in LLMs

TL;DR

This paper treats the capability–steerability relationship in large language models as an empirical question rather than an inevitability of higher capability. It distinguishes authorized versus unauthorized steerability and quantifies how minimal prompt interventions shift behavior using InstrumentalEval across Qwen3 variants, revealing steerability gaps on the order of tens of percentage points with a steerability gap $Δ$. The results show that short prompt suffixes can either amplify convergence or suppress it, with anti-instrumental prompts reducing convergence significantly for aligned models, while larger models may exhibit modest improvements in suppression alongside higher refusal rates. The work highlights a practical safety–security dilemma for open-weight models and discusses mitigation strategies such as unlearning, encrypted execution, and tamper-resistant checkpoints to balance control and misuse risk.

Abstract

We examine two properties of AI systems: capability (what a system can do) and steerability (how reliably one can shift behavior toward intended outcomes). A central question is whether capability growth reduces steerability and risks control collapse. We also distinguish between authorized steerability (builders reliably reaching intended behaviors) and unauthorized steerability (attackers eliciting disallowed behaviors). This distinction highlights a fundamental safety--security dilemma of AI models: safety requires high steerability to enforce control (e.g., stop/refuse), while security requires low steerability for malicious actors to elicit harmful behaviors. This tension presents a significant challenge for open-weight models, which currently exhibit high steerability via common techniques like fine-tuning or adversarial attacks. Using Qwen3 and InstrumentalEval, we find that a short anti-instrumental prompt suffix sharply reduces the measured convergence rate (e.g., shutdown avoidance, self-replication). For Qwen3-30B Instruct, the convergence rate drops from 81.69% under a pro-instrumental suffix to 2.82% under an anti-instrumental suffix. Under anti-instrumental prompting, larger aligned models show lower convergence rates than smaller ones (Instruct: 2.82% vs. 4.23%; Thinking: 4.23% vs. 9.86%). Code is available at github.com/j-hoscilowicz/instrumental_steering.