InnoGym: Benchmarking the Innovation Potential of AI Agents
LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both.
692ffb3026742347f61db108
Jintian Zhang
692ffb3026742347f61db109
Kewei Xu
692ffb3026742347f61db10a
Jingsheng Zheng
claimed_verified
2026-03-02T09:41:45.185Z
684bc1be17ae31ba66171292
https://cdn-avatars.huggingface.co/v1/production/uploads/684bc1be17ae31ba66171292/LFlkU4kArMjSzIbwjXd44.jpeg
Jingsheng Zheng
JohnsonZheng03
user
JohnsonZheng03
692ffb3026742347f61db10b
Zhuoyun Yu
692ffb3026742347f61db10c
Yuqi Zhu
692ffb3026742347f61db10d
Yujie Luo
692ffb3026742347f61db10e
Lanning Wei
692ffb3026742347f61db10f
Shuofei Qiao
claimed_verified
2026-02-18T09:06:15.355Z
6447800f30fa4ecb85ddad80
https://cdn-avatars.huggingface.co/v1/production/uploads/6447800f30fa4ecb85ddad80/NsmXIaMsWctmTNA7tFVkX.jpeg
Shuofei Qiao
GoooDte
user
GoooDte
692ffb3026742347f61db110
Lun Du
692ffb3026742347f61db111
Da Zheng
692ffb3026742347f61db112
Shumin Deng
692ffb3026742347f61db113
Huajun Chen
692ffb3026742347f61db114
Ningyu Zhang
claimed_verified
2025-12-04T08:48:35.607Z
620b3bbb0668e435407c8d0a
/avatars/e0fccbb2577d76088e09f054c35cffbc.svg
Ningyu Zhang
Ningyu
user
Ningyu
InnoGym: Benchmarking the Innovation Potential of AI Agents
LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both.
InnoGym is a benchmark and framework that evaluates the innovation potential of AI agents using performance gain and novelty metrics, highlighting a gap between creativity and effectiveness.
InnoGym performance gain novelty iGym reproducible long-horizon evaluations