How to Train Your LLM Web Agent: A Statistical Diagnosis
LLM-based web agents have recently made significant progress, but much of it
has occurred in closed-source systems, widening the gap with open-source
alternatives. Progress has been held back by two key challenges: first, a
narrow focus on single-step tasks that overlooks the complexity of multi-step
web interactions; and second, the high compute costs required to post-train
LLM-based web agents. To address this, we present the first statistically
grounded study on compute allocation for LLM web-agent post-training. Our
approach uses a two-stage pipeline, training a Llama 3.1 8B student to imitate
a Llama 3.3 70B teacher via supervised fine-tuning (SFT), followed by on-policy
reinforcement learning. We find this process highly sensitive to hyperparameter
choices, making exhaustive sweeps impractical. To spare others from expensive
trial-and-error, we sample 1,370 configurations and use bootstrapping to
estimate effective hyperparameters. Our results show that combining SFT with
on-policy RL consistently outperforms either approach alone on both WorkArena
and MiniWob++. Further, this strategy requires only 55% of the compute to match
the peak performance of pure SFT on MiniWob++, effectively pushing the
compute-performance Pareto frontier, and is the only strategy that can close
the gap with closed-source models.
686dbfe2cb5725779c60b314
Dheeraj Vattikonda
claimed_verified
2025-07-09T15:10:15.465Z
65fb7b1c876d4642e8340e95
/avatars/c258c4528315243b98c1e60ba398e98b.svg
Dheeraj Vattikonda
Dheeraj46329
user
Dheeraj46329
686dbfe2cb5725779c60b315
Santhoshi Ravichandran
686dbfe2cb5725779c60b316
Emiliano Penaloza
claimed_verified
2025-07-09T14:41:04.559Z
66c4d0fd18743b108e4e260e
https://cdn-avatars.huggingface.co/v1/production/uploads/66c4d0fd18743b108e4e260e/xKLb9t6VZCxt464VCkQ0O.png
Emiliano Penaloza
ppEmiliano
user
ppEmiliano
686dbfe2cb5725779c60b317
Hadi Nekoei
686dbfe2cb5725779c60b318
Megh Thakkar
686dbfe2cb5725779c60b319
Thibault Le Sellier de Chezelles
686dbfe2cb5725779c60b31a
Nicolas Gontier
686dbfe2cb5725779c60b31b
Miguel Muñoz-Mármol
686dbfe2cb5725779c60b31c
Sahar Omidi Shayegan
686dbfe2cb5725779c60b31d
Stefania Raimondo
686dbfe2cb5725779c60b31e
Xue Liu
686dbfe2cb5725779c60b31f
Alexandre Drouin
686dbfe2cb5725779c60b320
Laurent Charlin
686dbfe2cb5725779c60b321
Alexandre Piché
686dbfe2cb5725779c60b322
Alexandre Lacoste
686dbfe2cb5725779c60b323
Massimo Caccia
claimed_verified
2026-02-06T19:02:15.473Z
64b04320e5000ae8a57570a7
/avatars/226f57ea23b9dd861e71b7ba7821adce.svg
Massimo Caccia
optimass
user
optimass
How to Train Your LLM Web Agent: A Statistical Diagnosis
LLM-based web agents have recently made significant progress, but much of it
has occurred in closed-source systems, widening the gap with open-source
alternatives. Progress has been held back by two key challenges: first, a
narrow focus on single-step tasks that overlooks the complexity of multi-step
web interactions; and second, the high compute costs required to post-train
LLM-based web agents. To address this, we present the first statistically
grounded study on compute allocation for LLM web-agent post-training. Our
approach uses a two-stage pipeline, training a Llama 3.1 8B student to imitate
a Llama 3.3 70B teacher via supervised fine-tuning (SFT), followed by on-policy
reinforcement learning. We find this process highly sensitive to hyperparameter
choices, making exhaustive sweeps impractical. To spare others from expensive
trial-and-error, we sample 1,370 configurations and use bootstrapping to
estimate effective hyperparameters. Our results show that combining SFT with
on-policy RL consistently outperforms either approach alone on both WorkArena
and MiniWob++. Further, this strategy requires only 55% of the compute to match
the peak performance of pure SFT on MiniWob++, effectively pushing the
compute-performance Pareto frontier, and is the only strategy that can close
the gap with closed-source models.
A study on compute allocation for post-training LLM-based web agents finds that combining supervised fine-tuning with on-policy reinforcement learning improves performance and reduces computational costs compared to using either method alone.
LLM-based web agents supervised fine-tuning on-policy reinforcement learning hyperparameter choices bootstrapping WorkArena MiniWob++