Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning
Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. During RL, these models explore by generating new outputs, one token at a time. However, sampling actions token-by-token can result in highly inefficient learning, particularly when rewards are sparse. Here, we show that it is possible to overcome this problem by acting and exploring within the internal representations of an autoregressive model. Specifically, to discover temporally-abstract actions, we introduce a higher-order, non-causal sequence model whose outputs control the residual stream activations of a base autoregressive model. On grid world and MuJoCo-based tasks with hierarchical structure, we find that the higher-order model learns to compress long activation sequence chunks onto internal controllers. Critically, each controller executes a sequence of behaviorally meaningful actions that unfold over long timescales and are accompanied with a learned termination condition, such that composing multiple controllers over time leads to efficient exploration on novel tasks. We show that direct internal controller reinforcement, a process we term "internal RL", enables learning from sparse rewards in cases where standard RL finetuning fails. Our results demonstrate the benefits of latent action generation and reinforcement in autoregressive models, suggesting internal RL as a promising avenue for realizing hierarchical RL within foundation models.
694e76e5746a34b55dd545eb
Seijin Kobayashi
694e76e5746a34b55dd545ec
Yanick Schimpf
claimed_verified
2025-12-29T14:23:35.366Z
68288c78b4d8c06744817b87
/avatars/c46824aa788cbe358b322dfce04a59ed.svg
Yanick Schimpf
yschimpf
user
yschimpf
694e76e5746a34b55dd545ed
Maximilian Schlegel
claimed_verified
2025-12-29T14:23:37.525Z
67f3a5638a2b2e738c7aec2b
/avatars/f268121ab33da4c28789880d1086e7a5.svg
Maximilian Schlegel
schlegelm
user
schlegelm
694e76e5746a34b55dd545ee
Angelika Steger
694e76e5746a34b55dd545ef
Maciej Wolczyk
694e76e5746a34b55dd545f0
Johannes von Oswald
694e76e5746a34b55dd545f1
Nino Scherrer
694e76e5746a34b55dd545f2
Kaitlin Maile
694e76e5746a34b55dd545f3
Guillaume Lajoie
694e76e5746a34b55dd545f4
Blake A. Richards
694e76e5746a34b55dd545f5
Rif A. Saurous
694e76e5746a34b55dd545f6
James Manyika
694e76e5746a34b55dd545f7
Blaise Agüera y Arcas
694e76e5746a34b55dd545f8
Alexander Meulemans
694e76e5746a34b55dd545f9
João Sacramento
Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning
Large-scale autoregressive models pretrained on next-token prediction and finetuned with reinforcement learning (RL) have achieved unprecedented success on many problem domains. During RL, these models explore by generating new outputs, one token at a time. However, sampling actions token-by-token can result in highly inefficient learning, particularly when rewards are sparse. Here, we show that it is possible to overcome this problem by acting and exploring within the internal representations of an autoregressive model. Specifically, to discover temporally-abstract actions, we introduce a higher-order, non-causal sequence model whose outputs control the residual stream activations of a base autoregressive model. On grid world and MuJoCo-based tasks with hierarchical structure, we find that the higher-order model learns to compress long activation sequence chunks onto internal controllers. Critically, each controller executes a sequence of behaviorally meaningful actions that unfold over long timescales and are accompanied with a learned termination condition, such that composing multiple controllers over time leads to efficient exploration on novel tasks. We show that direct internal controller reinforcement, a process we term "internal RL", enables learning from sparse rewards in cases where standard RL finetuning fails. Our results demonstrate the benefits of latent action generation and reinforcement in autoregressive models, suggesting internal RL as a promising avenue for realizing hierarchical RL within foundation models.