SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce runtime while preserving program correctness. However, most benchmarks emphasize what to fix rather than how to fix code. We introduce SWE-fficiency, a benchmark for evaluating repository-level performance optimization on real workloads. Our suite contains 498 tasks across nine widely used data-science, machine-learning, and HPC repositories (e.g., numpy, pandas, scipy): given a complete codebase and a slow workload, an agent must investigate code semantics, localize bottlenecks and relevant tests, and produce a patch that matches or exceeds expert speedup while passing the same unit tests. To enable this how-to-fix evaluation, our automated pipeline scrapes GitHub pull requests for performance-improving edits, combining keyword filtering, static analysis, coverage tooling, and execution validation to both confirm expert speedup baselines and identify relevant repository unit tests. Empirical evaluation of state-of-the-art agents reveals significant underperformance. On average, agents achieve less than 0.15x the expert speedup: agents struggle in localizing optimization opportunities, reasoning about execution across functions, and maintaining correctness in proposed edits. We release the benchmark and accompanying data pipeline to facilitate research on automated performance engineering and long-horizon software reasoning.
6912b9f9a644ba07c499c789
Jeffrey Jian Ma
6912b9f9a644ba07c499c78a
Milad Hashemi
6912b9f9a644ba07c499c78b
Amir Yazdanbakhsh
6912b9f9a644ba07c499c78c
Kevin Swersky
6912b9f9a644ba07c499c78d
Ofir Press
6912b9f9a644ba07c499c78e
Enhui Li
6912b9f9a644ba07c499c78f
Vijay Janapa Reddi
6912b9f9a644ba07c499c790
Parthasarathy Ranganathan
SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?
Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce runtime while preserving program correctness. However, most benchmarks emphasize what to fix rather than how to fix code. We introduce SWE-fficiency, a benchmark for evaluating repository-level performance optimization on real workloads. Our suite contains 498 tasks across nine widely used data-science, machine-learning, and HPC repositories (e.g., numpy, pandas, scipy): given a complete codebase and a slow workload, an agent must investigate code semantics, localize bottlenecks and relevant tests, and produce a patch that matches or exceeds expert speedup while passing the same unit tests. To enable this how-to-fix evaluation, our automated pipeline scrapes GitHub pull requests for performance-improving edits, combining keyword filtering, static analysis, coverage tooling, and execution validation to both confirm expert speedup baselines and identify relevant repository unit tests. Empirical evaluation of state-of-the-art agents reveals significant underperformance. On average, agents achieve less than 0.15x the expert speedup: agents struggle in localizing optimization opportunities, reasoning about execution across functions, and maintaining correctness in proposed edits. We release the benchmark and accompanying data pipeline to facilitate research on automated performance engineering and long-horizon software reasoning.
SWE-fficiency is a benchmark for evaluating repository-level performance optimization using real workloads, focusing on identifying and implementing performance improvements while maintaining code correctness.
code reasoning software engineering performance optimization benchmark repository-level code semantics bottleneck localization unit tests automated pipeline GitHub pull requests static analysis coverage tooling execution validation expert speedup software reasoning