Latent Adversarial Regularization for Offline Preference Optimization
Learning from human feedback typically relies on preference optimization that constrains policy updates through token-level regularization. However, preference optimization for language models is particularly challenging because token-space similarity does not imply semantic or behavioral similarity. To address this challenge, we leverage latent-space regularization for language model preference optimization. We introduce GANPO, which achieves latent-space regularization by penalizing divergence between the internal representations of a policy model and a reference model. Given that latent representations are not associated with explicit probability densities, we adopt an adversarial approach inspired by GANs to minimize latent-space divergence. We integrate GANPO as a regularizer into existing offline preference optimization objectives. Experiments across multiple model architectures and tasks show consistent improvements from latent-space regularization. Further, by comparing GANPO-induced inferential biases with those from token-level regularization, we find that GANPO provides more robust structural feedback under distributional shift and noise while maintaining comparable downstream performance with minor computational overhead.
697cfaa26676f9332270616e
Enyi Jiang
claimed_verified
2026-02-02T16:58:36.305Z
6058351db2c84d5386b3afe5
https://cdn-avatars.huggingface.co/v1/production/uploads/6058351db2c84d5386b3afe5/mS1Nu9i5VYgAnHSKXz-b4.jpeg
Enyi (Olivia) Jiang
EnyiJiang
user
EnyiJiang
697cfaa26676f9332270616f
Yibo Jacky Zhang
697cfaa26676f93322706170
Yinglun Xu
697cfaa26676f93322706171
Andreas Haupt
697cfaa26676f93322706172
Nancy Amato
697cfaa26676f93322706173
Sanmi Koyejo
Latent Adversarial Regularization for Offline Preference Optimization
Learning from human feedback typically relies on preference optimization that constrains policy updates through token-level regularization. However, preference optimization for language models is particularly challenging because token-space similarity does not imply semantic or behavioral similarity. To address this challenge, we leverage latent-space regularization for language model preference optimization. We introduce GANPO, which achieves latent-space regularization by penalizing divergence between the internal representations of a policy model and a reference model. Given that latent representations are not associated with explicit probability densities, we adopt an adversarial approach inspired by GANs to minimize latent-space divergence. We integrate GANPO as a regularizer into existing offline preference optimization objectives. Experiments across multiple model architectures and tasks show consistent improvements from latent-space regularization. Further, by comparing GANPO-induced inferential biases with those from token-level regularization, we find that GANPO provides more robust structural feedback under distributional shift and noise while maintaining comparable downstream performance with minor computational overhead.
GANPO uses latent-space regularization through adversarial divergence minimization to improve language model preference optimization, offering more robust structural feedback than token-level methods.
preference optimization language models latent-space regularization token-level regularization policy model reference model adversarial approach GANs latent-space divergence offline preference optimization distributional shift computational overhead