Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization

Maksim Afanasyev, Illarion Iov

TL;DR

SLIME addresses a key gap in preference-based alignment: optimizing the relative margin can erode absolute generation quality. It introduces a three-part objective that anchors the chosen sequence likelihood, stabilizes rejected token probabilities, and employs a dual-margin mechanism to shape the decision boundary. Empirical results across multiple 3–4B parameter models and benchmarks show SLIME consistently outperforms DPO and SimPO, with reduced unlearning and improved generation stability. The work provides a practical pathway for robust, reference-free alignment and motivates further exploration of theoretical guarantees and online extensions.

Abstract

Direct preference optimization methods have emerged as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning Large Language Models (LLMs). Latest approaches have streamlined the alignment process by deriving implicit reward functions, yet they often suffer from a critical objective mismatch: optimizing the relative margin between chosen and rejected responses does not guarantee the preservation of the chosen response's absolute likelihood. This can lead to ``unlearning'', where the model degrades the probability of high-quality outputs to satisfy margin constraints, and ``formatting collapse'' caused by the over-penalization of rejected sequences. In this work, we introduce SLIME (Stabilized Likelihood Implicit Margin Enforcement), a reference-free alignment objective designed to decouple preference learning from generation quality. SLIME incorporates a three-pronged objective: (1) an anchoring term to maximize the likelihood of preferred responses; (2) a stabilizing penalty that prevents the probabilities of rejected tokens from collapsing to zero; and (3) a dual-margin mechanism that combines hard and soft constraints for precise boundary shaping. Our results demonstrate that SLIME achieves superior performance compared to state-of-the-art baselines while maintaining higher generation stability.

SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization

TL;DR

SLIME addresses a key gap in preference-based alignment: optimizing the relative margin can erode absolute generation quality. It introduces a three-part objective that anchors the chosen sequence likelihood, stabilizes rejected token probabilities, and employs a dual-margin mechanism to shape the decision boundary. Empirical results across multiple 3–4B parameter models and benchmarks show SLIME consistently outperforms DPO and SimPO, with reduced unlearning and improved generation stability. The work provides a practical pathway for robust, reference-free alignment and motivates further exploration of theoretical guarantees and online extensions.

Abstract

Direct preference optimization methods have emerged as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning Large Language Models (LLMs). Latest approaches have streamlined the alignment process by deriving implicit reward functions, yet they often suffer from a critical objective mismatch: optimizing the relative margin between chosen and rejected responses does not guarantee the preservation of the chosen response's absolute likelihood. This can lead to ``unlearning'', where the model degrades the probability of high-quality outputs to satisfy margin constraints, and ``formatting collapse'' caused by the over-penalization of rejected sequences. In this work, we introduce SLIME (Stabilized Likelihood Implicit Margin Enforcement), a reference-free alignment objective designed to decouple preference learning from generation quality. SLIME incorporates a three-pronged objective: (1) an anchoring term to maximize the likelihood of preferred responses; (2) a stabilizing penalty that prevents the probabilities of rejected tokens from collapsing to zero; and (3) a dual-margin mechanism that combines hard and soft constraints for precise boundary shaping. Our results demonstrate that SLIME achieves superior performance compared to state-of-the-art baselines while maintaining higher generation stability.
Paper Structure (39 sections, 32 equations, 5 tables)

This paper contains 39 sections, 32 equations, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Standard Alignment and Direct Methods
  4. Kahneman-Tversky Optimization (KTO)
  5. SimPO: Simple Preference Optimization
  6. Generalization via $\Psi$-Preference Optimization
  7. Group Relative Policy Optimization (GRPO)
  8. Sequence-level and Global Optimization
  9. Method
  10. Overview and Motivation
  11. Anchoring the Chosen Sequence
  12. Stabilizing Rejected Token Probabilities
  13. Dual-Margin Preference Optimization
  14. Experiments
  15. Experimental Setup
  16. ...and 24 more sections