Spatial Competence Benchmark
Spatial competence is the quality of maintaining a consistent internal representation of an environment and using it to infer discrete structure and plan actions under constraints. Prevailing spatial evaluations for large models are limited to probing isolated primitives through 3D transformations or visual question answering. We introduce the Spatial Competence Benchmark (SCBench), spanning three hierarchical capability buckets whose tasks require executable outputs verified by deterministic checkers or simulator-based evaluators. On SCBench, three frontier models exhibit monotonically decreasing accuracy up the capability ladder. Sweeping output-token caps shows that accuracy gains concentrate at low budgets and saturate quickly, and failures are dominated by locally plausible geometry that breaks global constraints. We release the task generators, verifiers, and visualisation tooling.
69de22259d30fe1923da4a37
Jash Vira
claimed_verified
2026-04-15T12:06:35.847Z
6655367680b530dba7c4ab46
/avatars/8d42164904b416c07c36eb5a140e50fb.svg
Jash Vira
jash404
user
jash404
69de22259d30fe1923da4a38
Ashley Harris
Spatial Competence Benchmark
Spatial competence is the quality of maintaining a consistent internal representation of an environment and using it to infer discrete structure and plan actions under constraints. Prevailing spatial evaluations for large models are limited to probing isolated primitives through 3D transformations or visual question answering. We introduce the Spatial Competence Benchmark (SCBench), spanning three hierarchical capability buckets whose tasks require executable outputs verified by deterministic checkers or simulator-based evaluators. On SCBench, three frontier models exhibit monotonically decreasing accuracy up the capability ladder. Sweeping output-token caps shows that accuracy gains concentrate at low budgets and saturate quickly, and failures are dominated by locally plausible geometry that breaks global constraints. We release the task generators, verifiers, and visualisation tooling.
Three frontier models show declining accuracy on a new spatial competence benchmark, with performance saturating quickly under token budget constraints.
spatial competence internal representation discrete structure action planning spatial evaluation capability buckets executable outputs deterministic checkers simulator-based evaluators token budget constraints