π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs
Flow-based vision-language-action (VLA) models excel in embodied control but suffer from intractable likelihoods during multi-step sampling, hindering online reinforcement learning. We propose \textit{boldsymbolπ-StepNFT} (Step-wise Negative-aware Fine-Tuning), a critic-and-likelihood-free framework that requires only a single forward pass per optimization step and eliminates auxiliary value networks. We identify that wider exploration spaces necessitate finer-grained, step-wise guidance for alignment. Empirically, π-StepNFT unlocks latent potential on LIBERO with competitive few-shot robustness. Moreover, it achieves superior generalization on ManiSkill, outperforming value-based baselines in OOD scenarios by preventing overfitting to multimodal features. This property offers a scalable solution promising for complex real-world applications.
69ae3b1a9893f58272f74105
Siting Wang
69ae3b1a9893f58272f74106
Xiaofeng Wang
claimed_verified
2026-03-09T08:46:52.664Z
6426616ea5ec4a5cbc535634
https://cdn-avatars.huggingface.co/v1/production/uploads/6426616ea5ec4a5cbc535634/5IfSFYd9QOxz8K9QmBCst.png
JeffWang
Jeff-Wang
user
Jeff-Wang
69ae3b1a9893f58272f74107
Zheng Zhu
69ae3b1a9893f58272f74108
Minnan Pei
69ae3b1a9893f58272f74109
Xinyu Cui
69ae3b1a9893f58272f7410a
Cheng Deng
69ae3b1a9893f58272f7410b
Jian Zhao
69ae3b1a9893f58272f7410c
Guan Huang
69ae3b1a9893f58272f7410d
Haifeng Zhang
69ae3b1a9893f58272f7410e
Jun Wang
π-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs
Flow-based vision-language-action (VLA) models excel in embodied control but suffer from intractable likelihoods during multi-step sampling, hindering online reinforcement learning. We propose \textit{boldsymbolπ-StepNFT} (Step-wise Negative-aware Fine-Tuning), a critic-and-likelihood-free framework that requires only a single forward pass per optimization step and eliminates auxiliary value networks. We identify that wider exploration spaces necessitate finer-grained, step-wise guidance for alignment. Empirically, π-StepNFT unlocks latent potential on LIBERO with competitive few-shot robustness. Moreover, it achieves superior generalization on ManiSkill, outperforming value-based baselines in OOD scenarios by preventing overfitting to multimodal features. This property offers a scalable solution promising for complex real-world applications.
Step-wise negative-aware fine-tuning enables efficient reinforcement learning for vision-language-action models by eliminating likelihood computation and auxiliary networks while improving generalization in complex environments.
flow-based vision-language-action models embodied control multi-step sampling reinforcement learning negative-aware fine-tuning critic-free framework optimization steps exploration spaces alignment few-shot robustness generalization out-of-distribution scenarios multimodal features scalable solutions