LTD-Bench: Evaluating Large Language Models by Letting Them Draw
Current evaluation paradigms for large language models (LLMs) represent a
critical blind spot in AI research--relying on opaque numerical metrics that
conceal fundamental limitations in spatial reasoning while providing no
intuitive understanding of model capabilities. This deficiency creates a
dangerous disconnect between reported performance and practical abilities,
particularly for applications requiring physical world understanding. We
introduce LTD-Bench, a breakthrough benchmark that transforms LLM evaluation
from abstract scores to directly observable visual outputs by requiring models
to generate drawings through dot matrices or executable code. This approach
makes spatial reasoning limitations immediately apparent even to non-experts,
bridging the fundamental gap between statistical performance and intuitive
assessment. LTD-Bench implements a comprehensive methodology with complementary
generation tasks (testing spatial imagination) and recognition tasks (assessing
spatial perception) across three progressively challenging difficulty levels,
methodically evaluating both directions of the critical language-spatial
mapping. Our extensive experiments with state-of-the-art models expose an
alarming capability gap: even LLMs achieving impressive results on traditional
benchmarks demonstrate profound deficiencies in establishing bidirectional
mappings between language and spatial concept--a fundamental limitation that
undermines their potential as genuine world models. Furthermore, LTD-Bench's
visual outputs enable powerful diagnostic analysis, offering a potential
approach to investigate model similarity.
690ad8c4d70e173c8452902c
Liuhao Lin
690ad8c4d70e173c8452902d
Ke Li
claimed_verified
2026-01-06T10:05:10.116Z
63280915eeee4dd858083092
/avatars/78347af4af42527d53e88d9969c5c934.svg
Ke Li
tristanli
user
tristanli
690ad8c4d70e173c8452902e
Zihan Xu
690ad8c4d70e173c8452902f
Yuchen Shi
claimed_verified
2026-01-04T20:32:31.435Z
622b00a776c20fee5d14501b
/avatars/e00496dda1e309548e7b5b437839bb65.svg
Eason shi
Easonshi
user
Easonshi
690ad8c4d70e173c84529030
Yulei Qin
claimed_verified
2025-11-05T16:24:35.496Z
6390525c00fb8ec4a424e0ff
/avatars/4302571e2ef4a9875491221aa630a329.svg
Yulei Qin
yolay
user
yolay
690ad8c4d70e173c84529031
Yan Zhang
690ad8c4d70e173c84529032
Xing Sun
claimed_verified
2025-11-27T10:15:27.980Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
tedsun
user
tedsun
690ad8c4d70e173c84529033
Rongrong Ji
LTD-Bench: Evaluating Large Language Models by Letting Them Draw
Current evaluation paradigms for large language models (LLMs) represent a
critical blind spot in AI research--relying on opaque numerical metrics that
conceal fundamental limitations in spatial reasoning while providing no
intuitive understanding of model capabilities. This deficiency creates a
dangerous disconnect between reported performance and practical abilities,
particularly for applications requiring physical world understanding. We
introduce LTD-Bench, a breakthrough benchmark that transforms LLM evaluation
from abstract scores to directly observable visual outputs by requiring models
to generate drawings through dot matrices or executable code. This approach
makes spatial reasoning limitations immediately apparent even to non-experts,
bridging the fundamental gap between statistical performance and intuitive
assessment. LTD-Bench implements a comprehensive methodology with complementary
generation tasks (testing spatial imagination) and recognition tasks (assessing
spatial perception) across three progressively challenging difficulty levels,
methodically evaluating both directions of the critical language-spatial
mapping. Our extensive experiments with state-of-the-art models expose an
alarming capability gap: even LLMs achieving impressive results on traditional
benchmarks demonstrate profound deficiencies in establishing bidirectional
mappings between language and spatial concept--a fundamental limitation that
undermines their potential as genuine world models. Furthermore, LTD-Bench's
visual outputs enable powerful diagnostic analysis, offering a potential
approach to investigate model similarity.
LTD-Bench evaluates large language models' spatial reasoning by requiring them to generate visual outputs, revealing significant limitations in their ability to map language to spatial concepts.
LTD-Bench large language models spatial reasoning dot matrices executable code spatial imagination spatial perception bidirectional mappings language-spatial mapping world models diagnostic analysis