Understanding and Improving Hyperbolic Deep Reinforcement Learning
The performance of reinforcement learning (RL) agents depends critically on the quality of the underlying feature representations. Hyperbolic feature spaces are well-suited for this purpose, as they naturally capture hierarchical and relational structure often present in complex RL environments. However, leveraging these spaces commonly faces optimization challenges due to the nonstationarity of RL. In this work, we identify key factors that determine the success and failure of training hyperbolic deep RL agents. By analyzing the gradients of core operations in the Poincaré Ball and Hyperboloid models of hyperbolic geometry, we show that large-norm embeddings destabilize gradient-based training, leading to trust-region violations in proximal policy optimization (PPO). Based on these insights, we introduce Hyper++, a new hyperbolic PPO agent that consists of three components: (i) stable critic training through a categorical value loss instead of regression; (ii) feature regularization guaranteeing bounded norms while avoiding the curse of dimensionality from clipping; and (iii) using a more optimization-friendly formulation of hyperbolic network layers. In experiments on ProcGen, we show that Hyper++ guarantees stable learning, outperforms prior hyperbolic agents, and reduces wall-clock time by approximately 30%. On Atari-5 with Double DQN, Hyper++ strongly outperforms Euclidean and hyperbolic baselines. We release our code at https://github.com/Probabilistic-and-Interactive-ML/hyper-rl .
6942c782fb33037a39577de8
Timo Klein
claimed_verified
2025-12-17T16:18:53.644Z
64ca3d44ec160b67ca5325e4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/drL9iiIpl8qN6qweKzsE4.jpeg
Timo K.
X3N4
user
X3N4
6942c782fb33037a39577de9
Thomas Lang
claimed_verified
2025-12-19T09:07:26.013Z
69440d3c9b62067cf59dc34f
/avatars/fff80249cf48d92c8a0ecee59114cc33.svg
Thomas Lang
langt94
user
langt94
6942c782fb33037a39577dea
Andrii Shkabrii
6942c782fb33037a39577deb
Alexander Sturm
6942c782fb33037a39577dec
Kevin Sidak
6942c782fb33037a39577ded
Lukas Miklautz
6942c782fb33037a39577dee
Claudia Plant
6942c782fb33037a39577def
Yllka Velaj
6942c782fb33037a39577df0
Sebastian Tschiatschek
Understanding and Improving Hyperbolic Deep Reinforcement Learning
The performance of reinforcement learning (RL) agents depends critically on the quality of the underlying feature representations. Hyperbolic feature spaces are well-suited for this purpose, as they naturally capture hierarchical and relational structure often present in complex RL environments. However, leveraging these spaces commonly faces optimization challenges due to the nonstationarity of RL. In this work, we identify key factors that determine the success and failure of training hyperbolic deep RL agents. By analyzing the gradients of core operations in the Poincaré Ball and Hyperboloid models of hyperbolic geometry, we show that large-norm embeddings destabilize gradient-based training, leading to trust-region violations in proximal policy optimization (PPO). Based on these insights, we introduce Hyper++, a new hyperbolic PPO agent that consists of three components: (i) stable critic training through a categorical value loss instead of regression; (ii) feature regularization guaranteeing bounded norms while avoiding the curse of dimensionality from clipping; and (iii) using a more optimization-friendly formulation of hyperbolic network layers. In experiments on ProcGen, we show that Hyper++ guarantees stable learning, outperforms prior hyperbolic agents, and reduces wall-clock time by approximately 30%. On Atari-5 with Double DQN, Hyper++ strongly outperforms Euclidean and hyperbolic baselines. We release our code at https://github.com/Probabilistic-and-Interactive-ML/hyper-rl .
Hyperbolic deep reinforcement learning agents face optimization challenges due to nonstationarity, but a new approach called Hyper++ addresses these issues through stable training methods and improved network formulations, demonstrating superior performance and efficiency.
reinforcement learning hyperbolic feature spaces Poincaré Ball Hyperboloid models proximal policy optimization trust-region violations categorical value loss feature regularization hyperbolic network layers ProcGen Atari-5 Double DQN