Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents
Large language models are increasingly deployed as autonomous agents executing multi-step workflows in real-world software environments. However, existing agent benchmarks suffer from three critical limitations: (1) trajectory-opaque grading that checks only final outputs, (2) underspecified safety and robustness evaluation, and (3) narrow modality coverage and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps. It comprises 300 human-verified tasks spanning 9 categories across three groups (general service orchestration, multimodal perception and generation, and multi-turn professional dialogue). Every agent action is recorded through three independent evidence channels (execution traces, audit logs, and environment snapshots), enabling trajectory-aware grading over 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, reporting Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models reveal that: (1) trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures that our hybrid pipeline catches; (2) controlled error injection primarily degrades consistency rather than peak capability, with Pass^3 dropping up to 24% while Pass@3 remains stable; (3) multimodal performance varies sharply, with most models performing poorer on video than on document or image, and no single model dominating across all modalities. Beyond benchmarking, Claw-Eval highlights actionable directions for agent development, shedding light on what it takes to build agents that are not only capable but reliably deployable.
69d5ba95b9b236b710ae80cc
Bowen Ye
claimed_verified
2026-04-09T18:51:03.291Z
663f2e2d9efa4d370061f6f6
/avatars/6e1391b66870f03b65c3fe346b6c37c0.svg
BoWen Ye
Ymiracle
user
Ymiracle
69d5ba95b9b236b710ae80cd
Rang Li
claimed_verified
2026-04-08T08:05:04.931Z
6847abd1892b779986a5ec80
/avatars/0f820eedf74d8e0636af94ef7cd4d04c.svg
Rang Li
lirang04
user
lirang04
69d5ba95b9b236b710ae80ce
Qibin Yang
69d5ba95b9b236b710ae80cf
Yuanxin Liu
claimed_verified
2026-04-08T16:34:14.964Z
6489761dcaea79f577897f98
/avatars/8f56dc9c08dc2b672555602d68509a03.svg
Yuanxin Liu
lyx97
user
lyx97
69d5ba95b9b236b710ae80d0
Linli Yao
69d5ba95b9b236b710ae80d1
Hanglong Lv
69d5ba95b9b236b710ae80d2
Zhihui Xie
69d5ba95b9b236b710ae80d3
Chenxin An
69d5ba95b9b236b710ae80d4
Lei Li
claimed_verified
2026-04-08T08:05:02.018Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
tobiaslee
user
tobiaslee
69d5ba95b9b236b710ae80d5
Lingpeng Kong
69d5ba95b9b236b710ae80d6
Qi Liu
69d5ba95b9b236b710ae80d7
Zhifang Sui
69d5ba95b9b236b710ae80d8
Tong Yang
Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents
Large language models are increasingly deployed as autonomous agents executing multi-step workflows in real-world software environments. However, existing agent benchmarks suffer from three critical limitations: (1) trajectory-opaque grading that checks only final outputs, (2) underspecified safety and robustness evaluation, and (3) narrow modality coverage and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps. It comprises 300 human-verified tasks spanning 9 categories across three groups (general service orchestration, multimodal perception and generation, and multi-turn professional dialogue). Every agent action is recorded through three independent evidence channels (execution traces, audit logs, and environment snapshots), enabling trajectory-aware grading over 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, reporting Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models reveal that: (1) trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures that our hybrid pipeline catches; (2) controlled error injection primarily degrades consistency rather than peak capability, with Pass^3 dropping up to 24% while Pass@3 remains stable; (3) multimodal performance varies sharply, with most models performing poorer on video than on document or image, and no single model dominating across all modalities. Beyond benchmarking, Claw-Eval highlights actionable directions for agent development, shedding light on what it takes to build agents that are not only capable but reliably deployable.
Claw-Eval addresses limitations in agent benchmarks by providing comprehensive evaluation across multiple modalities with trajectory-aware grading and safety assessments.
large language models autonomous agents multi-step workflows agent benchmarks trajectory-opaque grading safety evaluation robustness evaluation multimodal perception multimodal generation trajectory-aware grading Pass@k Pass^k