Understanding the Challenges in Iterative Generative Optimization with LLMs
Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make ``hidden'' design choices: What can the optimizer edit and what is the "right" learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices.
69c56a102de2ee20094ed966
Allen Nie
claimed_verified
2026-03-27T11:34:56.471Z
649466d90560480110a72247
/avatars/79fc060693328f4345c01adef4aba17c.svg
Allen Nie
allenanie
user
allenanie
69c56a102de2ee20094ed967
Xavier Daull
69c56a102de2ee20094ed968
Zhiyi Kuang
69c56a102de2ee20094ed969
Abhinav Akkiraju
69c56a102de2ee20094ed96a
Anish Chaudhuri
69c56a102de2ee20094ed96b
Max Piasevoli
69c56a102de2ee20094ed96c
Ryan Rong
69c56a102de2ee20094ed96d
YuCheng Yuan
69c56a102de2ee20094ed96e
Prerit Choudhary
69c56a102de2ee20094ed96f
Shannon Xiao
69c56a102de2ee20094ed970
Rasool Fakoor
69c56a102de2ee20094ed971
Adith Swaminathan
69c56a102de2ee20094ed972
Ching-An Cheng
Understanding the Challenges in Iterative Generative Optimization with LLMs
Generative optimization uses large language models (LLMs) to iteratively improve artifacts (such as code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make ``hidden'' design choices: What can the optimizer edit and what is the "right" learning evidence to provide at each update? We investigate three factors that affect most applications: the starting artifact, the credit horizon for execution traces, and batching trials and errors into learning evidence. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that these design decisions can determine whether generative optimization succeeds, yet they are rarely made explicit in prior work. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal way to set up learning loops across domains is a major hurdle for productionization and adoption. We provide practical guidance for making these choices.
Generative optimization using large language models faces challenges due to implicit design decisions about artifact modification and learning evidence that significantly impact success across different applications.
large language models generative optimization self-improving agents learning loops execution feedback automated optimization MLAgentBench Atari BigBench Extra Hard credit horizon minibatches