Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting
Offline post-training adapts a pretrained robot policy to a target dataset by supervised regression on recorded actions. In practice, robot datasets are heterogeneous: they mix embodiments, camera setups, and demonstrations of varying quality, so many trajectories reflect recovery behavior, inconsistent operator skill, or weakly informative supervision. Uniform post-training gives equal credit to all samples and can therefore average over conflicting or low-attribution data. We propose Posterior-Transition Reweighting (PTR), a reward-free and conservative post-training method that decides how much each training sample should influence the supervised update. For each sample, PTR encodes the observed post-action consequence as a latent target, inserts it into a candidate pool of mismatched targets, and uses a separate transition scorer to estimate a softmax identification posterior over target indices. The posterior-to-uniform ratio defines the PTR score, which is converted into a clipped-and-mixed weight and applied to the original action objective through self-normalized weighted regression. This construction requires no tractable policy likelihood and is compatible with both diffusion and flow-matching action heads. Rather than uniformly trusting all recorded supervision, PTR reallocates credit according to how attributable each sample's post-action consequence is under the current representation, improving conservative offline adaptation to heterogeneous robot data.
69bb83ebba2b62a5fb0b0d3c
Wanpeng Zhang
claimed_verified
2026-03-19T12:56:58.678Z
640dd700fdeaae139081f598
https://cdn-avatars.huggingface.co/v1/production/uploads/640dd700fdeaae139081f598/L986zu4-iOPFs9Y3_T5Ue.jpeg
Wanpeng Zhang
zawnpn
user
zawnpn
69bb83ebba2b62a5fb0b0d3d
Hao Luo
69bb83ebba2b62a5fb0b0d3e
Sipeng Zheng
69bb83ebba2b62a5fb0b0d3f
Yicheng Feng
69bb83ebba2b62a5fb0b0d40
Haiweng Xu
69bb83ebba2b62a5fb0b0d41
Ziheng Xi
69bb83ebba2b62a5fb0b0d42
Chaoyi Xu
69bb83ebba2b62a5fb0b0d43
Haoqi Yuan
69bb83ebba2b62a5fb0b0d44
Zongqing Lu
Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting
Offline post-training adapts a pretrained robot policy to a target dataset by supervised regression on recorded actions. In practice, robot datasets are heterogeneous: they mix embodiments, camera setups, and demonstrations of varying quality, so many trajectories reflect recovery behavior, inconsistent operator skill, or weakly informative supervision. Uniform post-training gives equal credit to all samples and can therefore average over conflicting or low-attribution data. We propose Posterior-Transition Reweighting (PTR), a reward-free and conservative post-training method that decides how much each training sample should influence the supervised update. For each sample, PTR encodes the observed post-action consequence as a latent target, inserts it into a candidate pool of mismatched targets, and uses a separate transition scorer to estimate a softmax identification posterior over target indices. The posterior-to-uniform ratio defines the PTR score, which is converted into a clipped-and-mixed weight and applied to the original action objective through self-normalized weighted regression. This construction requires no tractable policy likelihood and is compatible with both diffusion and flow-matching action heads. Rather than uniformly trusting all recorded supervision, PTR reallocates credit according to how attributable each sample's post-action consequence is under the current representation, improving conservative offline adaptation to heterogeneous robot data.
Posterior-Transition Reweighting (PTR) improves offline robot policy adaptation by dynamically weighting training samples based on the attribution of their post-action consequences, enabling more conservative and effective learning from heterogeneous datasets.
post-training supervised regression robot policy heterogeneous datasets uniform post-training reward-free conservative post-training latent target transition scorer softmax identification posterior self-normalized weighted regression diffusion action heads flow-matching action heads