Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the "sim2real" gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.
698e946fcace060ff123acee
Romain Froger
claimed_verified
2026-03-04T09:24:58.117Z
6463f29b31063f253c221c35
https://cdn-avatars.huggingface.co/v1/production/uploads/6463f29b31063f253c221c35/g1851skgauDpfFIJiRRHK.jpeg
Romain Froger
RomainFroger
user
RomainFroger
698e946fcace060ff123acef
Pierre Andrews
698e946fcace060ff123acf0
Matteo Bettini
698e946fcace060ff123acf1
Amar Budhiraja
698e946fcace060ff123acf2
Ricardo Silveira Cabral
698e946fcace060ff123acf3
Virginie Do
698e946fcace060ff123acf4
Emilien Garreau
698e946fcace060ff123acf5
Jean-Baptiste Gaya
698e946fcace060ff123acf6
Hugo Laurençon
698e946fcace060ff123acf7
Maxime Lecanu
698e946fcace060ff123acf8
Kunal Malkan
698e946fcace060ff123acf9
Dheeraj Mekala
698e946fcace060ff123acfa
Pierre Ménard
698e946fcace060ff123acfb
Gerard Moreno-Torres Bertran
698e946fcace060ff123acfc
Ulyana Piterbarg
698e946fcace060ff123acfd
Mikhail Plekhanov
698e946fcace060ff123acfe
Mathieu Rita
698e946fcace060ff123acff
Andrey Rusakov
698e946fcace060ff123ad00
Vladislav Vorotilov
698e946fcace060ff123ad01
Mengjue Wang
698e946fcace060ff123ad02
Ian Yu
698e946fcace060ff123ad03
Amine Benhalloum
698e946fcace060ff123ad04
Grégoire Mialon
698e946fcace060ff123ad05
Thomas Scialom
Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the "sim2real" gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.
Gaia2 presents a benchmark for evaluating large language model agents in asynchronous, dynamic environments with temporal constraints and multi-agent collaboration, featuring a write-action verifier for reinforcement learning and revealing trade-offs between reasoning, efficiency, and robustness.
large language model agents asynchronous environments temporal constraints multi-agent collaboration write-action verifier reinforcement learning from verifiable rewards pass@1 sim2real gap Agents Research Environments platform