Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. To account for finite-data approximation error in the learned safety boundary, we add a conformal prediction calibration step that adjusts the safety threshold and provides finite-sample probabilistic safety coverage. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.
69bed89b3fd3fef6237a64cd
Mumuksh Tayal
claimed_verified
2026-04-02T15:32:19.517Z
6251ea7fed957771b6fda22e
/avatars/a279a78d85dca1bb822b04a17002304b.svg
Mumuksh Tayal
MT16
user
MT16
69bed89b3fd3fef6237a64ce
Manan Tayal
claimed_verified
2026-03-23T09:53:19.371Z
68e5138abdb80c260c5cb17f
https://cdn-avatars.huggingface.co/v1/production/uploads/68e5138abdb80c260c5cb17f/DF-ITQ18HgfVnxU2gUGaC.png
Manan Tayal
tayalmanan
user
tayalmanan
69bed89b3fd3fef6237a64cf
Ravi Prakash
Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. To account for finite-data approximation error in the learned safety boundary, we add a conformal prediction calibration step that adjusts the safety threshold and provides finite-sample probabilistic safety coverage. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.
SafeFlow Q-Learning extends FQL to safe offline reinforcement learning by combining a Hamilton-Jacobi reachability-inspired safety value function with an efficient one-step flow policy, achieving lower inference latency and reduced constraint violations.
offline safe reinforcement learning Hamilton--Jacobi reachability safety value function Bellman recursion flow policy behavioral cloning one-step actor conformal prediction calibration safety threshold finite-sample probabilistic safety coverage