ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification.
69d61a5dc6ed9968ac5c65ab
Xiangyi Li
claimed_verified
2026-04-08T18:46:01.864Z
663fe2d26304d377fc253322
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/wuey_nNXSW4GthPYLfFS4.jpeg
Xiangyi Li
xdotli
user
xdotli
69d61a5dc6ed9968ac5c65ac
Kyoung Whan Choe
69d61a5dc6ed9968ac5c65ad
Yimin Liu
69d61a5dc6ed9968ac5c65ae
Xiaokun Chen
69d61a5dc6ed9968ac5c65af
Chujun Tao
69d61a5dc6ed9968ac5c65b0
Bingran You
claimed_verified
2026-04-10T07:45:16.354Z
663bd5fcfb931d4660fd18b7
https://cdn-avatars.huggingface.co/v1/production/uploads/663bd5fcfb931d4660fd18b7/S46UAXBRC3cifNCq8RAhK.jpeg
Bingran You
bingran-you
user
bingran-you
69d61a5dc6ed9968ac5c65b1
Wenbo Chen
claimed_verified
2026-04-09T08:07:10.102Z
669f476ab29f9fbc230928b7
https://cdn-avatars.huggingface.co/v1/production/uploads/669f476ab29f9fbc230928b7/sufiDwhj_DMx5SqhJ6UHX.jpeg
Wenbo Chen
wenbochen111
user
wenbochen111
69d61a5dc6ed9968ac5c65b2
Zonglin Di
69d61a5dc6ed9968ac5c65b3
Jiankai Sun
69d61a5dc6ed9968ac5c65b4
Shenghan Zheng
69d61a5dc6ed9968ac5c65b5
Jiajun Bao
69d61a5dc6ed9968ac5c65b6
Yuanli Wang
69d61a5dc6ed9968ac5c65b7
Weixiang Yan
69d61a5dc6ed9968ac5c65b8
Yiyuan Li
69d61a5dc6ed9968ac5c65b9
Han-chung Lee
ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification.
ClawsBench evaluates LLM agents in realistic productivity workflows using high-fidelity mock services and structured tasks, revealing varying task success and unsafe action rates across models and agent configurations.
large language model agents productivity tasks benchmark mock services state management deterministic snapshot/restore structured tasks agent scaffolding domain skills meta prompt task success rate unsafe action rate OpenClaw multi-step sandbox escalation silent contract modification