ECLIPTICA - A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment

TL;DR

This work introduces CITA (Contrastive Instruction-Tuned Alignment), combining SFT with contrastive preference optimization under an explicit geometric anchor to a reference model, which yields a stable Riemannian chart and keeps instruction updates within a shared neighborhood, so regimes stay nearby and traversable for reliable switching.

Abstract

Alignment in large language models (LLMs) is still largely static: after training, the policy is frozen. DPO, GRPO methods typically imprint one behavior into the weights, leaving little runtime control beyond prompt hacks or expensive re-alignment. We introduce ECLIPTICA, which treats alignment as instruction-driven and runtime-controllable: natural-language alignment instructions act as an explicit behavioral contract (stance, refusal boundary, verbosity) that modulates behavior on the fly under evolving safety requirements, user roles, and governance constraints. We introduce CITA (Contrastive Instruction-Tuned Alignment), combining SFT with contrastive preference optimization under an explicit geometric anchor to a reference model. This yields a stable Riemannian chart and keeps instruction updates within a shared neighborhood, so regimes stay nearby and traversable for reliable switching. To isolate policy switching from ordinary instruction following, we release the ECLIPTICA benchmark: 3000 controlled cases (300 prompts x 10 instruction types) where the user request is fixed and only the alignment instruction changes. On Llama-3.1-8B across five suites (ECLIPTICA, TruthfulQA, Conditional Safety, Length Control, LITMUS), CITA reaches 86.7% instruction-alignment efficiency, beating DPO (56.1%), GRPO (36.1%), and PPO (20.4%).

Figures (29)

• Figure 1: Teaser: Instruction-Conditioned Safety Alignment. Same prompt with different alignment instructions. Only CITA produces valid, coherent responses in both modes.
• Figure 2: ECLIPTICA instruction derivation pipeline. We synthesize candidate alignment instructions from five independent judge models, filter by semantic agreement (BERTScore), apply a two-rater quality gate, and compile a 10-instruction inventory used to instantiate the prompt-held-constant switching grid.
• Figure 3: CITA objective: contrastive preference learning under explicit alignment instructions. CITA trains on quadruples $(I,X,Y^+,Y^-)$ where the alignment instruction $I$defines the preference relation $Y^+\succ Y^-$. (1) Instruction-conditioned preference: the loss is a conditional logistic/contrastive objective on the score gap$\Delta_\theta=\log \pi_\theta(Y^+\!\mid I,X)-\log \pi_\theta(Y^-\!\mid I,X)$, with temperature $\beta$ controlling sharpness. (2) Mandatory KL anchor: a non-optional trust-region term ($\lambda>0$) constrains updates relative to a frozen reference $\pi_0$, stabilizing a switchable policy family$\{\pi_\theta(\cdot\mid I,\cdot)\}_{I\in\mathcal{I}}$ instead of collapsing to a single implicit regime. The gradient form highlights self-quenching updates via $(1-P^+)$: once the instruction-conditioned preference is satisfied, the preference force diminishes, improving reliability under switching.
• Figure 4: Training pipeline and comparators. All methods branch from SFT. PPO and GRPO are online methods requiring reward models/functions. DPO is an offline preference method. CITA stacks on preference optimization with explicit instruction-conditioning and a mandatory KL anchor.
• Figure 5: Preference reward margins.CITAInstruct ($\sim 7.5$) $>$CITANoInstruct ($\sim 7.2$) $>$DPO ($\sim 6.0$). Higher margins indicate sharper preference separation; the KL anchor is designed to prevent this separation from degenerating into a single dominant regime.