GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers
The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes bug discovery considerably harder for large language models (LLMs). In this paper, we take game development as a representative domain and introduce the Game Benchmark for Quality Assurance (GBQA), a benchmark containing 30 games and 124 human-verified bugs across three difficulty levels, to evaluate whether LLMs can autonomously detect software bugs. The benchmark is constructed using a multi-agent system that develops games and injects bugs in a scalable manner, with human experts in the loop to ensure correctness. Moreover, we provide a baseline interactive agent equipped with a multi-round ReAct loop and a memory mechanism, enabling long-horizon exploration of game environments for bug detection across different LLMs. Extensive experiments on frontier LLMs demonstrate that autonomous bug discovery remains highly challenging: the best-performing model, Claude-4.6-Opus in thinking mode, identifies only 48.39% of the verified bugs. We believe GBQA provides an adequate testbed and evaluation criterion, and that further progress on it will help close the gap in autonomous software engineering.
69d5e4a3b9b236b710ae8233
Shufan Jiang
claimed_verified
2026-04-08T07:42:48.650Z
659817704dbb962b7c724a61
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Pe3eKSx-hkkgYAvl_mGx0.png
Shufan Jiang
Tsumugii
user
Tsumugii
69d5e4a3b9b236b710ae8234
Chios Chen
69d5e4a3b9b236b710ae8235
Zhiyang Chen
GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers
The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes bug discovery considerably harder for large language models (LLMs). In this paper, we take game development as a representative domain and introduce the Game Benchmark for Quality Assurance (GBQA), a benchmark containing 30 games and 124 human-verified bugs across three difficulty levels, to evaluate whether LLMs can autonomously detect software bugs. The benchmark is constructed using a multi-agent system that develops games and injects bugs in a scalable manner, with human experts in the loop to ensure correctness. Moreover, we provide a baseline interactive agent equipped with a multi-round ReAct loop and a memory mechanism, enabling long-horizon exploration of game environments for bug detection across different LLMs. Extensive experiments on frontier LLMs demonstrate that autonomous bug discovery remains highly challenging: the best-performing model, Claude-4.6-Opus in thinking mode, identifies only 48.39% of the verified bugs. We believe GBQA provides an adequate testbed and evaluation criterion, and that further progress on it will help close the gap in autonomous software engineering.
Large language models struggle with autonomous bug discovery in complex runtime environments, as demonstrated by a new game development benchmark that reveals limited effectiveness of current approaches despite sophisticated multi-agent systems and interactive agents.
large language models autonomous bug discovery runtime environments multi-agent system ReAct loop memory mechanism game development software engineering