Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with
  Free-Form Preferences
Reward models (RMs) play a critical role in aligning AI behaviors with human
preferences, yet they face two fundamental challenges: (1) Modality Imbalance,
where most RMs are mainly focused on text and image modalities, offering
limited support for video, audio, and other modalities; and (2) Preference
Rigidity, where training on fixed binary preference pairs fails to capture the
complexity and diversity of personalized preferences. To address the above
challenges, we propose Omni-Reward, a step toward generalist omni-modal reward
modeling with support for free-form preferences, consisting of: (1) Evaluation:
We introduce Omni-RewardBench, the first omni-modal RM benchmark with free-form
preferences, covering nine tasks across five modalities including text, image,
video, audio, and 3D; (2) Data: We construct Omni-RewardData, a multimodal
preference dataset comprising 248K general preference pairs and 69K
instruction-tuning pairs for training generalist omni-modal RMs; (3) Model: We
propose Omni-RewardModel, which includes both discriminative and generative
RMs, and achieves strong performance on Omni-RewardBench as well as other
widely used reward modeling benchmarks.
69002f8622d452aac6dd42f3
Zhuoran Jin
claimed_verified
2025-12-25T20:54:38.520Z
643379416c6ecd58798421b3
/avatars/831db7eab2663abc33b176cf386b02f2.svg
Zhuoran Jin
jinzhuoran
user
jinzhuoran
69002f8622d452aac6dd42f4
Hongbang Yuan
69002f8622d452aac6dd42f5
Kejian Zhu
69002f8622d452aac6dd42f6
Jiachun Li
claimed_verified
2026-02-12T20:29:12.707Z
64b89dfa6a68a9a715df407e
https://cdn-avatars.huggingface.co/v1/production/uploads/64b89dfa6a68a9a715df407e/FpBAdClhr-oVAv11Bjwjs.jpeg
Jiachun Li
Septzzz
user
Septzzz
69002f8622d452aac6dd42f7
Pengfei Cao
69002f8622d452aac6dd42f8
Yubo Chen
69002f8622d452aac6dd42f9
Kang Liu
69002f8622d452aac6dd42fa
Jun Zhao
Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with
  Free-Form Preferences
Reward models (RMs) play a critical role in aligning AI behaviors with human
preferences, yet they face two fundamental challenges: (1) Modality Imbalance,
where most RMs are mainly focused on text and image modalities, offering
limited support for video, audio, and other modalities; and (2) Preference
Rigidity, where training on fixed binary preference pairs fails to capture the
complexity and diversity of personalized preferences. To address the above
challenges, we propose Omni-Reward, a step toward generalist omni-modal reward
modeling with support for free-form preferences, consisting of: (1) Evaluation:
We introduce Omni-RewardBench, the first omni-modal RM benchmark with free-form
preferences, covering nine tasks across five modalities including text, image,
video, audio, and 3D; (2) Data: We construct Omni-RewardData, a multimodal
preference dataset comprising 248K general preference pairs and 69K
instruction-tuning pairs for training generalist omni-modal RMs; (3) Model: We
propose Omni-RewardModel, which includes both discriminative and generative
RMs, and achieves strong performance on Omni-RewardBench as well as other
widely used reward modeling benchmarks.
Omni-Reward addresses modality imbalance and preference rigidity in reward models by introducing a benchmark, dataset, and model that support multiple modalities and free-form preferences.
reward models modality imbalance preference rigidity Omni-Reward Omni-RewardBench Omni-RewardData Omni-RewardModel discriminative RMs generative RMs