OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models
AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language World Models (LWMs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LWM-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks.
69de26629d30fe1923da4a46
Xiaomeng Hu
claimed_verified
2026-04-15T12:06:32.586Z
611a2731f2560d2024b4afd2
/avatars/294fa218d43cd206fbc2a6c49ef38820.svg
huxiaomeng
gregH
user
gregH
69de26629d30fe1923da4a47
Yinger Zhang
69de26629d30fe1923da4a48
Fei Huang
69de26629d30fe1923da4a49
Jianhong Tu
69de26629d30fe1923da4a4a
Yang Su
69de26629d30fe1923da4a4b
Lianghao Deng
69de26629d30fe1923da4a4c
Yuxuan Liu
69de26629d30fe1923da4a4d
Yantao Liu
69de26629d30fe1923da4a4e
Dayiheng Liu
69de26629d30fe1923da4a4f
Tsung-Yi Ho
OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language World Models
AI agents are expected to perform professional work across hundreds of occupational domains (from emergency department triage to nuclear reactor safety monitoring to customs import processing), yet existing benchmarks can only evaluate agents in the few domains where public environments exist. We introduce OccuBench, a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains, enabled by Language World Models (LWMs) that simulate domain-specific environments through LLM-driven tool response generation. Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity. OccuBench evaluates agents along two complementary dimensions: task completion across professional domains and environmental robustness under controlled fault injection (explicit errors, implicit data degradation, and mixed faults). We evaluate 15 frontier models across 8 model families and find that: (1) no single model dominates all industries, as each has a distinct occupational capability profile; (2) implicit faults (truncated data, missing fields) are harder than both explicit errors (timeouts, 500s) and mixed faults, because they lack overt error signals and require the agent to independently detect data degradation; (3) larger models, newer generations, and higher reasoning effort consistently improve performance. GPT-5.2 improves by 27.5 points from minimal to maximum reasoning effort; and (4) strong agents are not necessarily strong environment simulators. Simulator quality is critical for LWM-based evaluation reliability. OccuBench provides the first systematic cross-industry evaluation of AI agents on professional occupational tasks.
OccuBench presents a comprehensive benchmark for evaluating AI agents across 100 professional domains using Language World Models to simulate real-world environments with controlled fault injection.
Language World Models multi-agent synthesis pipeline task completion environmental robustness fault injection implicit faults explicit errors mixed faults reasoning effort simulator quality