SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In
  Text-only LLMs
Recent advances in text-only large language models (LLMs), such as
DeepSeek-R1, demonstrate remarkable reasoning ability. However, these models
remain fragile or entirely incapable when extended to multi-modal tasks.
Existing approaches largely rely on single-form captions, which lack diversity
and often fail to adapt across different types of Visual Question Answering
(VQA) benchmarks. As a result, they provide no principled or efficient channel
for transmitting fine-grained visual information. We introduce Seeing Eye, a
modular framework that unlocks multimodal reasoning in text-only LLMs through
an agent-based small VLM translator. This translator acts as a perception
agent: it can invoke specialized tools (e.g., OCR and crop) and iteratively
distill multimodal inputs into structured intermediate representations (SIRs)
tailored to the question. These SIRs are then passed to the text-only LLM,
which serves as a reasoning agent. Crucially, the translator and reasoner
engage in multi-round feedback and interaction, enabling the extraction of
targeted visual details and yielding more confident answers. Experiments on
knowledge-intensive VQA benchmarks, including MMMU and MIA-Bench, demonstrate
that Seeing Eye not only reduces inference cost but also surpasses much larger
end-to-end VLMs. For example, an instantiation combining a 3B-parameter vision
translator with an 8B-parameter language reasoner outperforms a monolithic 32B
VLM on challenging knowledge-based questions. Our results highlight that
decoupling perception from reasoning via agent information flow offers a
scalable and plug-and-play pathway to multimodal reasoning, allowing strong
text-only LLMs to fully leverage their reasoning capabilities. Code is
available at: https://github.com/ulab-uiuc/SeeingEye
6902dac372739622ee92a88b
Weijia Zhang
admin_assigned
2026-01-29T20:42:19.440Z
63a96be8c847db253f3cab95
https://cdn-avatars.huggingface.co/v1/production/uploads/63a96be8c847db253f3cab95/PRK9_qzCvVlI-LpvSaz6s.png
CharlieDreemur
CharlieDreemur
user
CharlieDreemur
6902dac372739622ee92a88c
Zijia Liu
claimed_verified
2025-10-30T14:16:41.637Z
65d188a4aa309d842e438ef1
https://cdn-avatars.huggingface.co/v1/production/uploads/65d188a4aa309d842e438ef1/t5zmBIytp-xnMoHUfYhEa.jpeg
Zijia Liu
m-serious
user
m-serious
6902dac372739622ee92a88d
Haoru Li
6902dac372739622ee92a88e
Haoqi Chen
6902dac372739622ee92a88f
Jiaxuan You
SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In
  Text-only LLMs
Recent advances in text-only large language models (LLMs), such as
DeepSeek-R1, demonstrate remarkable reasoning ability. However, these models
remain fragile or entirely incapable when extended to multi-modal tasks.
Existing approaches largely rely on single-form captions, which lack diversity
and often fail to adapt across different types of Visual Question Answering
(VQA) benchmarks. As a result, they provide no principled or efficient channel
for transmitting fine-grained visual information. We introduce Seeing Eye, a
modular framework that unlocks multimodal reasoning in text-only LLMs through
an agent-based small VLM translator. This translator acts as a perception
agent: it can invoke specialized tools (e.g., OCR and crop) and iteratively
distill multimodal inputs into structured intermediate representations (SIRs)
tailored to the question. These SIRs are then passed to the text-only LLM,
which serves as a reasoning agent. Crucially, the translator and reasoner
engage in multi-round feedback and interaction, enabling the extraction of
targeted visual details and yielding more confident answers. Experiments on
knowledge-intensive VQA benchmarks, including MMMU and MIA-Bench, demonstrate
that Seeing Eye not only reduces inference cost but also surpasses much larger
end-to-end VLMs. For example, an instantiation combining a 3B-parameter vision
translator with an 8B-parameter language reasoner outperforms a monolithic 32B
VLM on challenging knowledge-based questions. Our results highlight that
decoupling perception from reasoning via agent information flow offers a
scalable and plug-and-play pathway to multimodal reasoning, allowing strong
text-only LLMs to fully leverage their reasoning capabilities. Code is
available at: https://github.com/ulab-uiuc/SeeingEye
Seeing Eye, a modular framework, enhances multimodal reasoning in text-only LLMs by decoupling perception and reasoning through an agent-based translator and reasoner, outperforming larger end-to-end models on VQA benchmarks.
DeepSeek-R1 multimodal reasoning text-only LLMs agent-based small VLM translator perception agent specialized tools OCR crop structured intermediate representations SIRs reasoning agent multi-round feedback MMMU MIA-Bench vision translator language reasoner monolithic VLM