SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization
Direct preference optimization methods have emerged as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning Large Language Models (LLMs). Latest approaches have streamlined the alignment process by deriving implicit reward functions, yet they often suffer from a critical objective mismatch: optimizing the relative margin between chosen and rejected responses does not guarantee the preservation of the chosen response's absolute likelihood. This can lead to ``unlearning'', where the model degrades the probability of high-quality outputs to satisfy margin constraints, and ``formatting collapse'' caused by the over-penalization of rejected sequences. In this work, we introduce SLIME (Stabilized Likelihood Implicit Margin Enforcement), a reference-free alignment objective designed to decouple preference learning from generation quality. SLIME incorporates a three-pronged objective: (1) an anchoring term to maximize the likelihood of preferred responses; (2) a stabilizing penalty that prevents the probabilities of rejected tokens from collapsing to zero; and (3) a dual-margin mechanism that combines hard and soft constraints for precise boundary shaping. Our results demonstrate that SLIME achieves superior performance compared to state-of-the-art baselines while maintaining higher generation stability.
6981bfa3ce18b1862809650a
Maksim Afanasyev
claimed_verified
2026-02-03T10:02:16.327Z
62da74bf4be126e22e8cb549
https://cdn-avatars.huggingface.co/v1/production/uploads/62da74bf4be126e22e8cb549/eLZtsdGHuMP89M3Q0nCob.webp
Maksim Afanasyev
mrapplexz
user
mrapplexz
6981bfa3ce18b1862809650b
Illarion Iov
claimed_verified
2026-02-03T13:58:13.594Z
652108d0bafd014bf684bbbc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/UvdNMfUixBonZxCn5eOYK.jpeg
Illarion
DeadSailor
user
DeadSailor
SLIME: Stabilized Likelihood Implicit Margin Enforcement for Preference Optimization
Direct preference optimization methods have emerged as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning Large Language Models (LLMs). Latest approaches have streamlined the alignment process by deriving implicit reward functions, yet they often suffer from a critical objective mismatch: optimizing the relative margin between chosen and rejected responses does not guarantee the preservation of the chosen response's absolute likelihood. This can lead to ``unlearning'', where the model degrades the probability of high-quality outputs to satisfy margin constraints, and ``formatting collapse'' caused by the over-penalization of rejected sequences. In this work, we introduce SLIME (Stabilized Likelihood Implicit Margin Enforcement), a reference-free alignment objective designed to decouple preference learning from generation quality. SLIME incorporates a three-pronged objective: (1) an anchoring term to maximize the likelihood of preferred responses; (2) a stabilizing penalty that prevents the probabilities of rejected tokens from collapsing to zero; and (3) a dual-margin mechanism that combines hard and soft constraints for precise boundary shaping. Our results demonstrate that SLIME achieves superior performance compared to state-of-the-art baselines while maintaining higher generation stability.
SLIME is a novel reference-free alignment objective for large language models that decouples preference learning from generation quality through a three-pronged approach combining likelihood maximization, probability stabilization, and dual-margin constraints.
direct preference optimization Reinforcement Learning from Human Feedback RLHF Large Language Models LLMs implicit reward functions objective mismatch relative margin absolute likelihood unlearning formatting collapse reference-free alignment anchoring term stabilizing penalty dual-margin mechanism generation stability