Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal
  Reasoning in MLLMs
While Multimodal Large Language Models (MLLMs) excel at visual understanding,
they often struggle in complex scenarios that require visual planning and
imagination. Inspired by how humans use sketching as a form of visual thinking
to develop and communicate ideas, we introduce Latent Sketchpad, a framework
that equips MLLMs with an internal visual scratchpad. The internal visual
representations of MLLMs have traditionally been confined to perceptual
understanding. We repurpose them to support generative visual thought without
compromising reasoning ability. Building on frontier MLLMs, our approach
integrates visual generation directly into their native autoregressive
reasoning process. It allows the model to interleave textual reasoning with the
generation of visual latents. These latents guide the internal thought process
and can be translated into sketch images for interpretability. To realize this,
we introduce two components: a Context-Aware Vision Head autoregressively
produces visual representations, and a pretrained Sketch Decoder renders these
into human-interpretable images. We evaluate the framework on our new dataset
MazePlanning. Experiments across various MLLMs show that Latent Sketchpad
delivers comparable or even superior reasoning performance to their backbone.
It further generalizes across distinct frontier MLLMs, including Gemma3 and
Qwen2.5-VL. By extending model's textual reasoning to visual thinking, our
framework opens new opportunities for richer human-computer interaction and
broader applications. More details and resources are available on our project
page: https://latent-sketchpad.github.io/.
690173b7646208eac0d1f36c
Huanyu Zhang
690173b7646208eac0d1f36d
Wenshan Wu
690173b7646208eac0d1f36e
Chengzu Li
690173b7646208eac0d1f36f
Ning Shang
690173b7646208eac0d1f370
Yan Xia
690173b7646208eac0d1f371
Yangyu Huang
690173b7646208eac0d1f372
Yifan Zhang
690173b7646208eac0d1f373
Li Dong
claimed_verified
2025-10-31T14:28:18.087Z
5df85abada6d0311fd3d5408
/avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
Li Dong
unilm
user
unilm
690173b7646208eac0d1f374
Zhang Zhang
690173b7646208eac0d1f375
Liang Wang
690173b7646208eac0d1f376
Tieniu Tan
690173b7646208eac0d1f377
Furu Wei
Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal
  Reasoning in MLLMs
While Multimodal Large Language Models (MLLMs) excel at visual understanding,
they often struggle in complex scenarios that require visual planning and
imagination. Inspired by how humans use sketching as a form of visual thinking
to develop and communicate ideas, we introduce Latent Sketchpad, a framework
that equips MLLMs with an internal visual scratchpad. The internal visual
representations of MLLMs have traditionally been confined to perceptual
understanding. We repurpose them to support generative visual thought without
compromising reasoning ability. Building on frontier MLLMs, our approach
integrates visual generation directly into their native autoregressive
reasoning process. It allows the model to interleave textual reasoning with the
generation of visual latents. These latents guide the internal thought process
and can be translated into sketch images for interpretability. To realize this,
we introduce two components: a Context-Aware Vision Head autoregressively
produces visual representations, and a pretrained Sketch Decoder renders these
into human-interpretable images. We evaluate the framework on our new dataset
MazePlanning. Experiments across various MLLMs show that Latent Sketchpad
delivers comparable or even superior reasoning performance to their backbone.
It further generalizes across distinct frontier MLLMs, including Gemma3 and
Qwen2.5-VL. By extending model's textual reasoning to visual thinking, our
framework opens new opportunities for richer human-computer interaction and
broader applications. More details and resources are available on our project
page: https://latent-sketchpad.github.io/.
Latent Sketchpad enhances Multimodal Large Language Models with an internal visual scratchpad, enabling generative visual thought and improved reasoning performance.
Multimodal Large Language Models Latent Sketchpad visual planning imagination sketching visual thinking internal visual scratchpad perceptual understanding generative visual thought autoregressive reasoning visual latents sketch images Context-Aware Vision Head Sketch Decoder MazePlanning Gemma3 Qwen2.5-VL human-computer interaction