CodeClash: Benchmarking Goal-Oriented Software Engineering
Current benchmarks for coding evaluate language models (LMs) on concrete,
well-specified tasks such as fixing specific bugs or writing targeted tests.
However, human programmers do not spend all day incessantly addressing isolated
tasks. Instead, real-world software development is grounded in the pursuit of
high-level goals, like improving user retention or reducing costs. Evaluating
whether LMs can also iteratively develop code to better accomplish open-ended
objectives without any explicit guidance remains an open challenge. To address
this, we introduce CodeClash, a benchmark where LMs compete in multi-round
tournaments to build the best codebase for achieving a competitive objective.
Each round proceeds in two phases: agents edit their code, then their codebases
compete head-to-head in a code arena that determines winners based on
objectives like score maximization, resource acquisition, or survival. Whether
it's writing notes, scrutinizing documentation, analyzing competition logs, or
creating test suites, models must decide for themselves how to improve their
codebases both absolutely and against their opponents. We run 1680 tournaments
(25,200 rounds total) to evaluate 8 LMs across 6 arenas. Our results reveal
that while models exhibit diverse development styles, they share fundamental
limitations in strategic reasoning. Models also struggle with long-term
codebase maintenance, as repositories become progressively messy and redundant.
These limitations are stark: top models lose every round against expert human
programmers. We open-source CodeClash to advance the study of autonomous,
goal-oriented code development.
690b07cdd70e173c845290c4
John Yang
claimed_verified
2025-11-05T09:30:15.256Z
677c2e5032582d07139f8af1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ZxdbXnPAFq_O20FAZuD4v.png
John
john-b-yang
user
john-b-yang
690b07cdd70e173c845290c5
Kilian Lieret
690b07cdd70e173c845290c6
Joyce Yang
690b07cdd70e173c845290c7
Carlos E. Jimenez
690b07cdd70e173c845290c8
Ofir Press
690b07cdd70e173c845290c9
Ludwig Schmidt
690b07cdd70e173c845290ca
Diyi Yang
CodeClash: Benchmarking Goal-Oriented Software Engineering
Current benchmarks for coding evaluate language models (LMs) on concrete,
well-specified tasks such as fixing specific bugs or writing targeted tests.
However, human programmers do not spend all day incessantly addressing isolated
tasks. Instead, real-world software development is grounded in the pursuit of
high-level goals, like improving user retention or reducing costs. Evaluating
whether LMs can also iteratively develop code to better accomplish open-ended
objectives without any explicit guidance remains an open challenge. To address
this, we introduce CodeClash, a benchmark where LMs compete in multi-round
tournaments to build the best codebase for achieving a competitive objective.
Each round proceeds in two phases: agents edit their code, then their codebases
compete head-to-head in a code arena that determines winners based on
objectives like score maximization, resource acquisition, or survival. Whether
it's writing notes, scrutinizing documentation, analyzing competition logs, or
creating test suites, models must decide for themselves how to improve their
codebases both absolutely and against their opponents. We run 1680 tournaments
(25,200 rounds total) to evaluate 8 LMs across 6 arenas. Our results reveal
that while models exhibit diverse development styles, they share fundamental
limitations in strategic reasoning. Models also struggle with long-term
codebase maintenance, as repositories become progressively messy and redundant.
These limitations are stark: top models lose every round against expert human
programmers. We open-source CodeClash to advance the study of autonomous,
goal-oriented code development.
CodeClash evaluates language models' ability to iteratively develop code for open-ended objectives through competitive multi-round tournaments.
language models CodeClash multi-round tournaments codebase strategic reasoning long-term codebase maintenance