TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
While Vision-Language-Action (VLA) models have seen rapid progress in pretraining, their advancement in Reinforcement Learning (RL) remains hampered by low sample efficiency and sparse rewards in real-world settings. Developing generalizable process reward models is essential for providing the fine-grained feedback necessary to bridge this gap, yet existing temporal value functions often fail to generalize beyond their training domains. We introduce TOPReward, a novel, probabilistically grounded temporal value function that leverages the latent world knowledge of pretrained video Vision-Language Models (VLMs) to estimate robotic task progress. Unlike prior methods that prompt VLMs to directly output progress values, which are prone to numerical misrepresentation, TOPReward extracts task progress directly from the VLM's internal token logits. In zero-shot evaluations across 130+ distinct real-world tasks and multiple robot platforms (e.g., Franka, YAM, SO-100/101), TOPReward achieves 0.947 mean Value-Order Correlation (VOC) on Qwen3-VL, dramatically outperforming the state-of-the-art GVL baseline which achieves near-zero correlation on the same open-source model. We further demonstrate that TOPReward serves as a versatile tool for downstream applications, including success detection and reward-aligned behavior cloning.
699d1f754e37ec6dfa1bc5ff
Shirui Chen
699d1f754e37ec6dfa1bc600
Cole Harrison
699d1f754e37ec6dfa1bc601
Ying-Chun Lee
claimed_verified
2026-03-23T11:00:56.066Z
65df2bb4cec6b1667f120cad
https://cdn-avatars.huggingface.co/v1/production/uploads/65df2bb4cec6b1667f120cad/TWJJ6vXIYAUlcegHVlTJd.jpeg
Ying-Chun Lee
thomas0829
user
thomas0829
699d1f754e37ec6dfa1bc602
Angela Jin Yang
699d1f754e37ec6dfa1bc603
Zhongzheng Ren
699d1f754e37ec6dfa1bc604
Lillian J. Ratliff
699d1f754e37ec6dfa1bc605
Jiafei Duan
699d1f754e37ec6dfa1bc606
Dieter Fox
699d1f754e37ec6dfa1bc607
Ranjay Krishna
TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics
While Vision-Language-Action (VLA) models have seen rapid progress in pretraining, their advancement in Reinforcement Learning (RL) remains hampered by low sample efficiency and sparse rewards in real-world settings. Developing generalizable process reward models is essential for providing the fine-grained feedback necessary to bridge this gap, yet existing temporal value functions often fail to generalize beyond their training domains. We introduce TOPReward, a novel, probabilistically grounded temporal value function that leverages the latent world knowledge of pretrained video Vision-Language Models (VLMs) to estimate robotic task progress. Unlike prior methods that prompt VLMs to directly output progress values, which are prone to numerical misrepresentation, TOPReward extracts task progress directly from the VLM's internal token logits. In zero-shot evaluations across 130+ distinct real-world tasks and multiple robot platforms (e.g., Franka, YAM, SO-100/101), TOPReward achieves 0.947 mean Value-Order Correlation (VOC) on Qwen3-VL, dramatically outperforming the state-of-the-art GVL baseline which achieves near-zero correlation on the same open-source model. We further demonstrate that TOPReward serves as a versatile tool for downstream applications, including success detection and reward-aligned behavior cloning.
TOPReward is a probabilistically grounded temporal value function that uses pretrained video Vision-Language Models to estimate robotic task progress through internal token logits, achieving superior performance in zero-shot evaluations across diverse real-world tasks.
Vision-Language-Action models Reinforcement Learning temporal value functions Vision-Language Models token logits Value-Order Correlation reward-aligned behavior cloning