Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AntiPaSTO: Self-Supervised Steering of Moral Reasoning

Michael J. Clark

TL;DR

AntiPaSTO presents a self-supervised, internal steering method that operates on transformer representations to modulate moral reasoning without using preference labels. It trains an anti-parallel steering direction in the SVD space of residual-writer weights via incomplete contrast prefixes, coupled with coherence and monotonicity losses, enabling bidirectional control and out-of-distribution transfer to unseen dilemmas. The approach yields substantial gains over prompting baselines and can bypass common output-level refusals, though it exhibits seed- and scale-related sensitivity and requires careful hyperparameter exploration. This work advances gradient-based internal steering, provides a practical framework for studying controllability in alignment research, and releases code for safety-oriented investigations.

Abstract

As models grow more capable, human supervision breaks down: labels don't scale, outputs can be gamed, and training doesn't generalize. Scalable oversight requires steering methods that are internal, self-supervised, and transfer out-of-distribution; existing methods satisfy some but not all three. We introduce AntiPaSTO, which separates representations along an anti-parallel axis ($α=\pm1$ produce opposite shifts), with coherence constraints preventing collapse. Human input is minimal: two contrasting words inserted into template sentences, no preference labels. Using 800 such pairs on Gemma-3-1B, AntiPaSTO beats prompting baselines by $6.9\times$ on DailyDilemmas and maintains bidirectional control where prompting triggers refusal. Code is available at https://github.com/wassname/AntiPaSTO.

AntiPaSTO: Self-Supervised Steering of Moral Reasoning

TL;DR

AntiPaSTO presents a self-supervised, internal steering method that operates on transformer representations to modulate moral reasoning without using preference labels. It trains an anti-parallel steering direction in the SVD space of residual-writer weights via incomplete contrast prefixes, coupled with coherence and monotonicity losses, enabling bidirectional control and out-of-distribution transfer to unseen dilemmas. The approach yields substantial gains over prompting baselines and can bypass common output-level refusals, though it exhibits seed- and scale-related sensitivity and requires careful hyperparameter exploration. This work advances gradient-based internal steering, provides a practical framework for studying controllability in alignment research, and releases code for safety-oriented investigations.

Abstract

As models grow more capable, human supervision breaks down: labels don't scale, outputs can be gamed, and training doesn't generalize. Scalable oversight requires steering methods that are internal, self-supervised, and transfer out-of-distribution; existing methods satisfy some but not all three. We introduce AntiPaSTO, which separates representations along an anti-parallel axis ( produce opposite shifts), with coherence constraints preventing collapse. Human input is minimal: two contrasting words inserted into template sentences, no preference labels. Using 800 such pairs on Gemma-3-1B, AntiPaSTO beats prompting baselines by on DailyDilemmas and maintains bidirectional control where prompting triggers refusal. Code is available at https://github.com/wassname/AntiPaSTO.
Paper Structure (57 sections, 1 theorem, 15 equations, 4 figures, 12 tables)

This paper contains 57 sections, 1 theorem, 15 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Problem Setup
  4. Method
  5. Contrastive Data
  6. Representation Refinement
  7. Loss
  8. Projection ($\mathcal{L}_{\text{proj}}$).
  9. Coherence region constraint ($\mathcal{B}_{\text{coh}}$).
  10. Monotonicity constraint ($\mathcal{B}_{\text{mono}}$).
  11. Adapter
  12. Summary of key components.
  13. Results
  14. Evaluation Setup.
  15. Steering F1.
  16. ...and 42 more sections

Key Result

Proposition 1.1

Let $\text{TV}(p_{\text{steer}}(\cdot|c), p_{\text{ref}}(\cdot|c)) \leq \theta_c$ for all contexts $c$ in the training distribution. Then:

Figures (4)

  • Figure 1: Bidirectional control test on a moral dilemma. Left: Persona prompting fails---the model refuses to roleplay dishonesty. Right: AntiPaSTO with $\alpha=\pm1$ produces opposite answers using the same adapter, demonstrating reliable bidirectional control.
  • Figure 2: Incomplete contrast pairs. (a) Two prefixes differ by one persona word. (b) If completed, trajectories would diverge---but we stop before generation. (c) Representations are $\sim$95% identical; the difference $\Delta h = h_{\text{cho}} - h_{\text{rej}}$ is small. (d) Since trajectories would branch differently, the branching information must be encoded in $\Delta h$. This is the self-supervised training signal: no completions, no preference labels.
  • Figure 3: Anti-parallel projection loss geometry. The loss trains $\delta_+$ (shift at $\alpha=+1$) and $\delta_-$ (shift at $\alpha=-1$) to align anti-parallel along $d_{\text{ref}}$. Left: Before training, shifts are random. Right: After training, $\delta_+$ aligns with $d_{\text{ref}}$ and $\delta_-$ anti-aligns, giving $\cos(\delta_+, d_{\text{ref}}) \times \cos(\delta_-, d_{\text{ref}}) < 0$. Dashed circle: coherence bound.
  • Figure 4: AntiPaSTO adapter architecture. Activations are projected into SVD space, rotated via learnable Cayley transforms, scaled by coefficient-dependent singular value perturbations, and projected back to activation space.

Theorems & Definitions (1)

  • Proposition 1.1: Coherence Transfer