Physics Informed Viscous Value Representations
Offline goal-conditioned reinforcement learning (GCRL) learns goal-conditioned policies from static pre-collected datasets. However, accurate value estimation remains a challenge due to the limited coverage of the state-action space. Recent physics-informed approaches have sought to address this by imposing physical and geometric constraints on the value function through regularization defined over first-order partial differential equations (PDEs), such as the Eikonal equation. However, these formulations can often be ill-posed in complex, high-dimensional environments. In this work, we propose a physics-informed regularization derived from the viscosity solution of the Hamilton-Jacobi-Bellman (HJB) equation. By providing a physics-based inductive bias, our approach grounds the learning process in optimal control theory, explicitly regularizing and bounding updates during value iterations. Furthermore, we leverage the Feynman-Kac theorem to recast the PDE solution as an expectation, enabling a tractable Monte Carlo estimation of the objective that avoids numerical instability in higher-order gradients. Experiments demonstrate that our method improves geometric consistency, making it broadly applicable to navigation and high-dimensional, complex manipulation tasks. Open-source codes are available at https://github.com/HrishikeshVish/phys-fk-value-GCRL.
69af146eb3ab31dee1281a86
Hrishikesh Viswanath
claimed_verified
2026-03-10T09:45:37.072Z
6740bf94d6d1d9c3b9fcf887
/avatars/ef0c0b056622b83784237803eb1521a1.svg
Hrishikesh Viswanath
hrishivish23
user
hrishivish23
69af146eb3ab31dee1281a87
Juanwu Lu
69af146eb3ab31dee1281a88
S. Talha Bukhari
69af146eb3ab31dee1281a89
Damon Conover
69af146eb3ab31dee1281a8a
Ziran Wang
69af146eb3ab31dee1281a8b
Aniket Bera
Physics Informed Viscous Value Representations
Offline goal-conditioned reinforcement learning (GCRL) learns goal-conditioned policies from static pre-collected datasets. However, accurate value estimation remains a challenge due to the limited coverage of the state-action space. Recent physics-informed approaches have sought to address this by imposing physical and geometric constraints on the value function through regularization defined over first-order partial differential equations (PDEs), such as the Eikonal equation. However, these formulations can often be ill-posed in complex, high-dimensional environments. In this work, we propose a physics-informed regularization derived from the viscosity solution of the Hamilton-Jacobi-Bellman (HJB) equation. By providing a physics-based inductive bias, our approach grounds the learning process in optimal control theory, explicitly regularizing and bounding updates during value iterations. Furthermore, we leverage the Feynman-Kac theorem to recast the PDE solution as an expectation, enabling a tractable Monte Carlo estimation of the objective that avoids numerical instability in higher-order gradients. Experiments demonstrate that our method improves geometric consistency, making it broadly applicable to navigation and high-dimensional, complex manipulation tasks. Open-source codes are available at https://github.com/HrishikeshVish/phys-fk-value-GCRL.
Physics-informed regularization based on Hamilton-Jacobi-Bellman equation with viscosity solution improves value estimation in offline goal-conditioned reinforcement learning through Monte Carlo estimation.
goal-conditioned reinforcement learning physics-informed regularization Hamilton-Jacobi-Bellman equation viscosity solution Eikonal equation value function optimal control theory Feynman-Kac theorem Monte Carlo estimation value iteration