Meta-Harness: End-to-End Optimization of Model Harnesses
The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.
69ccf5ab1f6782218d973f19
Yoonho Lee
69ccf5ab1f6782218d973f1a
Roshen Nair
69ccf5ab1f6782218d973f1b
Qizheng Zhang
claimed_verified
2026-04-13T07:35:22.133Z
6663e5b10c54dffcd2a921ca
/avatars/2a4589fef05306ccde06728c752e5601.svg
Qizheng Zhang
qizhengz
user
qizhengz
69ccf5ab1f6782218d973f1c
Kangwook Lee
69ccf5ab1f6782218d973f1d
Omar Khattab
69ccf5ab1f6782218d973f1e
Chelsea Finn
Meta-Harness: End-to-End Optimization of Model Harnesses
The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering.
Meta-Harness automates the design of code frameworks for large language models by searching over harness code, demonstrating superior performance in text classification, math reasoning, and agentic coding tasks.
large language model text optimizer context management system agentic proposer outer-loop system harness code execution traces retrieval-augmented math reasoning agentic coding TerminalBench-2