GameDevBench: Evaluating Agentic Capabilities Through Game Development
Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. Game development provides such a testbed as agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 132 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex -- the average solution requires over three times the amount of lines of code and file changes compared to prior software development benchmarks. Agents still struggle with game development, with the best agent solving only 54.5% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with success rates dropping from 46.9% on gameplay-oriented tasks to 31.6% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, with the largest change being an increase in Claude Sonnet 4.5's performance from 33.3% to 47.7%. We release GameDevBench publicly to support further research into agentic game development.
698d4fd565c0d15a6d1621cb
Wayne Chi
698d4fd565c0d15a6d1621cc
Yixiong Fang
698d4fd565c0d15a6d1621cd
Arnav Yayavaram
698d4fd565c0d15a6d1621ce
Siddharth Yayavaram
698d4fd565c0d15a6d1621cf
Seth Karten
claimed_verified
2026-02-17T15:50:49.927Z
6658e1c8ce1b2838885b2d7f
/avatars/8623555f14b62f40fd372da20cb59ccc.svg
Seth Karten
milkkarten
user
milkkarten
698d4fd565c0d15a6d1621d0
Qiuhong Anna Wei
698d4fd565c0d15a6d1621d1
Runkun Chen
698d4fd565c0d15a6d1621d2
Alexander Wang
698d4fd565c0d15a6d1621d3
Valerie Chen
698d4fd565c0d15a6d1621d4
Ameet Talwalkar
698d4fd565c0d15a6d1621d5
Chris Donahue
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. Game development provides such a testbed as agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 132 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex -- the average solution requires over three times the amount of lines of code and file changes compared to prior software development benchmarks. Agents still struggle with game development, with the best agent solving only 54.5% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with success rates dropping from 46.9% on gameplay-oriented tasks to 31.6% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, with the largest change being an increase in Claude Sonnet 4.5's performance from 33.3% to 47.7%. We release GameDevBench publicly to support further research into agentic game development.
GameDevBench is introduced as the first benchmark for evaluating agents on game development tasks that combine software development complexity with deep multimodal understanding requirements.
coding agents multimodal counterparts evaluation testbed software development multimodal understanding game development GameDevBench web tutorials video tutorials multimodal complexity agentic game development