WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG
Dynamical systems theory and reinforcement learning view world evolution as latent-state dynamics driven by actions, with visual observations providing partial information about the state. Recent video world models attempt to learn this action-conditioned dynamics from data. However, existing datasets rarely match the requirement: they typically lack diverse and semantically meaningful action spaces, and actions are directly tied to visual observations rather than mediated by underlying states. As a result, actions are often entangled with pixel-level changes, making it difficult for models to learn structured world dynamics and maintain consistent evolution over long horizons. In this paper, we propose WildWorld, a large-scale action-conditioned world modeling dataset with explicit state annotations, automatically collected from a photorealistic AAA action role-playing game (Monster Hunter: Wilds). WildWorld contains over 108 million frames and features more than 450 actions, including movement, attacks, and skill casting, together with synchronized per-frame annotations of character skeletons, world states, camera poses, and depth maps. We further derive WildBench to evaluate models through Action Following and State Alignment. Extensive experiments reveal persistent challenges in modeling semantically rich actions and maintaining long-horizon state consistency, highlighting the need for state-aware video generation. The project page is https://shandaai.github.io/wildworld-project/.
69c343b795b48f16a3007764
Zhen Li
claimed_verified
2026-03-26T14:04:10.645Z
685269199f20805782107cfb
/avatars/3a11c7658ccd7ad5b7cdf30afe3d3b31.svg
Zhen Li
Lixsp11
user
Lixsp11
69c343b795b48f16a3007765
Zian Meng
69c343b795b48f16a3007766
Shuwei Shi
69c343b795b48f16a3007767
Wenshuo Peng
69c343b795b48f16a3007768
Yuwei Wu
69c343b795b48f16a3007769
Bo Zheng
69c343b795b48f16a300776a
Chuanhao Li
claimed_verified
2026-03-27T11:38:14.274Z
674ea59a8f2e7614a6c72f26
/avatars/86fd4c6d7d33435de49b35659bf65265.svg
Chuanhao
ChuanhaoLi
user
ChuanhaoLi
69c343b795b48f16a300776b
Kaipeng Zhang
claimed_verified
2026-03-25T10:50:43.253Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
kpzhang996
user
kpzhang996
WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG
Dynamical systems theory and reinforcement learning view world evolution as latent-state dynamics driven by actions, with visual observations providing partial information about the state. Recent video world models attempt to learn this action-conditioned dynamics from data. However, existing datasets rarely match the requirement: they typically lack diverse and semantically meaningful action spaces, and actions are directly tied to visual observations rather than mediated by underlying states. As a result, actions are often entangled with pixel-level changes, making it difficult for models to learn structured world dynamics and maintain consistent evolution over long horizons. In this paper, we propose WildWorld, a large-scale action-conditioned world modeling dataset with explicit state annotations, automatically collected from a photorealistic AAA action role-playing game (Monster Hunter: Wilds). WildWorld contains over 108 million frames and features more than 450 actions, including movement, attacks, and skill casting, together with synchronized per-frame annotations of character skeletons, world states, camera poses, and depth maps. We further derive WildBench to evaluate models through Action Following and State Alignment. Extensive experiments reveal persistent challenges in modeling semantically rich actions and maintaining long-horizon state consistency, highlighting the need for state-aware video generation. The project page is https://shandaai.github.io/wildworld-project/.
WildWorld is a large-scale dataset for action-conditioned world modeling that provides explicit state annotations from a photorealistic game, enabling better understanding of latent-state dynamics and long-horizon consistency.
dynamical systems theory reinforcement learning latent-state dynamics action-conditioned dynamics video world models action spaces visual observations state annotations photorealistic AAA action role-playing game per-frame annotations character skeletons world states camera poses depth maps Action Following State Alignment long-horizon state consistency state-aware video generation