Spaces:
Running on CPU Upgrade
Open source traces
Could you open source the traces? You mentioned this in your page
⚠️ All scores on this page are self reported. Associated traces are made available to the open-source community in order to enable deeper study of the tradeoffs between model behavior vs performance on Gaia2.
We used to store them automatically as datasets when I coded the app but it's been entirely redone by people at Meta before release, let me ping @gregmialz who might know what's happening there now
Thanks @clefourrier , any progress. In particular, I am seeing lower performances than reported for Claude Sonnet.
This would be very useful if the released traces also include a compact operational receipt per run, not only the model-facing trajectory.
The fields I keep wanting for agent evals are:
- run id / parent run id
- task and allowed scope
- agent scaffold/version
- tool calls with normalized action class
- state changes or artifacts written
- checks run or skipped
- retry / escalation / stop reason
- final outcome and evaluator verdict
That makes the traces more comparable across agent scaffolds. It also lets people evaluate behavior like “used the right kind of tool”, “changed the expected state”, or “stopped cleanly”, not just whether the final answer matched.
I’m exploring this receipt shape while building Armorer/Guard, so I’d be very interested in how GAIA2 traces end up representing this: https://github.com/ArmorerLabs/Armorer/discussions/43#discussioncomment-17046441