Self-Improving World Modelling with Latent Actions
Internal modelling of the world -- predicting transitions between previous states X and next states Y under actions Z -- is essential to reasoning and planning for LLMs and VLMs. Learning such models typically requires costly action-labelled trajectories. We propose SWIRL, a self-improvement framework that learns from state-only sequences by treating actions as a latent variable and alternating between Forward World Modelling (FWM) P_θ(Y|X,Z) and an Inverse Dynamics Modelling (IDM) Q_φ(Z|X,Y). SWIRL iterates two phases: (1) Variational Information Maximisation, which updates the FWM to generate next states that maximise conditional mutual information with latent actions given prior states, encouraging identifiable consistency; and (2) ELBO Maximisation, which updates the IDM to explain observed transitions, effectively performing coordinate ascent. Both models are trained with reinforcement learning (specifically, GRPO) with the opposite frozen model's log-probability as a reward signal. We provide theoretical learnability guarantees for both updates, and evaluate SWIRL on LLMs and VLMs across multiple environments: single-turn and multi-turn open-world visual dynamics and synthetic textual environments for physics, web, and tool calling. SWIRL achieves gains of 16% on AURORABench, 28% on ByteMorph, 16% on WorldPredictionBench, and 14% on StableToolBench.
6989cb0bbeecc443208d2864
Yifu Qiu
6989cb0bbeecc443208d2865
Zheng Zhao
claimed_verified
2026-02-09T21:06:19.908Z
64ba8e9d5299e0f164491e45
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba8e9d5299e0f164491e45/ZK4bUJZuUO8xsj0RZmzMn.jpeg
Zheng Zhao
zsquaredz
user
zsquaredz
6989cb0bbeecc443208d2866
Waylon Li
claimed_verified
2026-02-10T09:27:43.273Z
63dcfaaaf37111482522fbd6
https://cdn-avatars.huggingface.co/v1/production/uploads/63dcfaaaf37111482522fbd6/-NkJznxAJum4fMPy8zoUT.jpeg
Waylon Li
waylonli
user
waylonli
6989cb0bbeecc443208d2867
Yftah Ziser
6989cb0bbeecc443208d2868
Anna Korhonen
6989cb0bbeecc443208d2869
Shay B. Cohen
6989cb0bbeecc443208d286a
Edoardo M. Ponti
Self-Improving World Modelling with Latent Actions
Internal modelling of the world -- predicting transitions between previous states X and next states Y under actions Z -- is essential to reasoning and planning for LLMs and VLMs. Learning such models typically requires costly action-labelled trajectories. We propose SWIRL, a self-improvement framework that learns from state-only sequences by treating actions as a latent variable and alternating between Forward World Modelling (FWM) P_θ(Y|X,Z) and an Inverse Dynamics Modelling (IDM) Q_φ(Z|X,Y). SWIRL iterates two phases: (1) Variational Information Maximisation, which updates the FWM to generate next states that maximise conditional mutual information with latent actions given prior states, encouraging identifiable consistency; and (2) ELBO Maximisation, which updates the IDM to explain observed transitions, effectively performing coordinate ascent. Both models are trained with reinforcement learning (specifically, GRPO) with the opposite frozen model's log-probability as a reward signal. We provide theoretical learnability guarantees for both updates, and evaluate SWIRL on LLMs and VLMs across multiple environments: single-turn and multi-turn open-world visual dynamics and synthetic textual environments for physics, web, and tool calling. SWIRL achieves gains of 16% on AURORABench, 28% on ByteMorph, 16% on WorldPredictionBench, and 14% on StableToolBench.
SWIRL is a self-improvement framework that learns world models from state-only sequences by alternating between forward and inverse dynamics modeling with variational information maximization and ELBO maximization, achieving improved performance on various reasoning and planning benchmarks.
Forward World Modelling Inverse Dynamics Modelling variational information maximisation ELBO maximisation reinforcement learning GRPO latent variables conditional mutual information coordinate ascent world models state-only sequences