The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
We present the PokeAgent Challenge, a large-scale benchmark for decision-making research built on Pokemon's multi-agent battle system and expansive role-playing game (RPG) environment. Partial observability, game-theoretic reasoning, and long-horizon planning remain open problems for frontier AI, yet few benchmarks stress all three simultaneously under realistic conditions. PokeAgent targets these limitations at scale through two complementary tracks: our Battling Track, which calls for strategic reasoning and generalization under partial observability in competitive Pokemon battles, and our Speedrunning Track, which requires long-horizon planning and sequential decision-making in the Pokemon RPG. Our Battling Track supplies a dataset of 20M+ battle trajectories alongside a suite of heuristic, RL, and LLM-based baselines capable of high-level competitive play. Our Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons of harness-based LLM approaches. Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokemon, with over 100 teams competing across both tracks and winning solutions detailed in our paper. Participant submissions and our baselines reveal considerable gaps between generalist (LLM), specialist (RL), and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokemon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing suites and positioning Pokemon as an unsolved benchmark that can drive RL and LLM research forward. We transition to a living benchmark with a live leaderboard for Battling and self-contained evaluation for Speedrunning at https://pokeagentchallenge.com.
69b9803febc57f8925831f6b
Seth Karten
claimed_verified
2026-03-17T17:33:44.411Z
6658e1c8ce1b2838885b2d7f
/avatars/8623555f14b62f40fd372da20cb59ccc.svg
Seth Karten
milkkarten
user
milkkarten
69b9803febc57f8925831f6c
Jake Grigsby
69b9803febc57f8925831f6d
Tersoo Upaa
69b9803febc57f8925831f6e
Junik Bae
69b9803febc57f8925831f6f
Seonghun Hong
69b9803febc57f8925831f70
Hyunyoung Jeong
69b9803febc57f8925831f71
Jaeyoon Jung
claimed_verified
2026-03-18T08:46:00.000Z
646484cfb90150b2706df03b
https://cdn-avatars.huggingface.co/v1/production/uploads/646484cfb90150b2706df03b/8ocSbXBSbrruhlhcxwzEt.png
Jaeyoon Jung
lastdefiance20
user
lastdefiance20
69b9803febc57f8925831f72
Kun Kerdthaisong
claimed_verified
2026-03-18T08:46:01.796Z
63a83c5432ed73936eb8363e
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/7GsqlWkTRy0kjLy5CdYIZ.png
kun kerdthaisong
augustus2011
user
augustus2011
69b9803febc57f8925831f73
Gyungbo Kim
69b9803febc57f8925831f74
Hyeokgi Kim
69b9803febc57f8925831f75
Yujin Kim
69b9803febc57f8925831f76
Eunju Kwon
69b9803febc57f8925831f77
Dongyu Liu
69b9803febc57f8925831f78
Patrick Mariglia
69b9803febc57f8925831f79
Sangyeon Park
69b9803febc57f8925831f7a
Benedikt Schink
69b9803febc57f8925831f7b
Xianwei Shi
69b9803febc57f8925831f7c
Anthony Sistilli
69b9803febc57f8925831f7d
Joseph Twin
69b9803febc57f8925831f7e
Arian Urdu
69b9803febc57f8925831f7f
Matin Urdu
69b9803febc57f8925831f80
Qiao Wang
69b9803febc57f8925831f81
Ling Wu
69b9803febc57f8925831f82
Wenli Zhang
69b9803febc57f8925831f83
Kunsheng Zhou
69b9803febc57f8925831f84
Stephanie Milani
69b9803febc57f8925831f85
Kiran Vodrahalli
69b9803febc57f8925831f86
Amy Zhang
69b9803febc57f8925831f87
Fei Fang
69b9803febc57f8925831f88
Yuke Zhu
69b9803febc57f8925831f89
Chi Jin
The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
We present the PokeAgent Challenge, a large-scale benchmark for decision-making research built on Pokemon's multi-agent battle system and expansive role-playing game (RPG) environment. Partial observability, game-theoretic reasoning, and long-horizon planning remain open problems for frontier AI, yet few benchmarks stress all three simultaneously under realistic conditions. PokeAgent targets these limitations at scale through two complementary tracks: our Battling Track, which calls for strategic reasoning and generalization under partial observability in competitive Pokemon battles, and our Speedrunning Track, which requires long-horizon planning and sequential decision-making in the Pokemon RPG. Our Battling Track supplies a dataset of 20M+ battle trajectories alongside a suite of heuristic, RL, and LLM-based baselines capable of high-level competitive play. Our Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons of harness-based LLM approaches. Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokemon, with over 100 teams competing across both tracks and winning solutions detailed in our paper. Participant submissions and our baselines reveal considerable gaps between generalist (LLM), specialist (RL), and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokemon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing suites and positioning Pokemon as an unsolved benchmark that can drive RL and LLM research forward. We transition to a living benchmark with a live leaderboard for Battling and self-contained evaluation for Speedrunning at https://pokeagentchallenge.com.
The PokeAgent Challenge establishes a large-scale benchmark for decision-making research using Pokemon's multi-agent battle system and RPG environment, featuring two tracks that test strategic reasoning, partial observability, and long-horizon planning in competitive and speedrunning contexts.
multi-agent battle system role-playing game partial observability game-theoretic reasoning long-horizon planning large-scale benchmark Battling Track Speedrunning Track battle trajectories heuristic baselines RL baselines LLM baselines competitive play RPG speedrunning multi-agent orchestration system harness-based LLM approaches NeurIPS competition BenchPress evaluation matrix living benchmark live leaderboard