Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models
Scaling up multimodal models has enabled remarkable advances in visual understanding and reasoning, but practical demands call for smaller, efficient systems. In this work, we conduct a principled analysis of downscaling intelligence in multimodal models, examining how reduced large language model (LLM) capacity affects multimodal capabilities. Our initial findings reveal an interesting trend: LLM downscaling disproportionately affects visual capabilities, rather than abilities inherited from the LLM. We then examine whether this drop mainly reflects the expected decline in visual reasoning or a more fundamental loss of perceptual abilities. Isolating the effect of LLM downscaling on perception, we find performance still drops sharply, often matching or exceeding the impact on reasoning. To address this bottleneck, we introduce visual extraction tuning, which explicitly trains the model to extract instruction-relevant visual details consistently across tasks. With these extracted visual details, we then apply step-by-step reasoning to generate answers. Together, these components form our Extract+Think approach, setting a new standard for efficiency and performance in this space.
69240fbcb5612535ed955a5e
Mark Endo
claimed_verified
2025-11-24T12:26:50.910Z
65c4287b6b3715a9cf28ded9
/avatars/2af7d2c64665cb5283398084628c1701.svg
Mark Endo
markendo
user
markendo
69240fbcb5612535ed955a5f
Serena Yeung-Levy
Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models
Scaling up multimodal models has enabled remarkable advances in visual understanding and reasoning, but practical demands call for smaller, efficient systems. In this work, we conduct a principled analysis of downscaling intelligence in multimodal models, examining how reduced large language model (LLM) capacity affects multimodal capabilities. Our initial findings reveal an interesting trend: LLM downscaling disproportionately affects visual capabilities, rather than abilities inherited from the LLM. We then examine whether this drop mainly reflects the expected decline in visual reasoning or a more fundamental loss of perceptual abilities. Isolating the effect of LLM downscaling on perception, we find performance still drops sharply, often matching or exceeding the impact on reasoning. To address this bottleneck, we introduce visual extraction tuning, which explicitly trains the model to extract instruction-relevant visual details consistently across tasks. With these extracted visual details, we then apply step-by-step reasoning to generate answers. Together, these components form our Extract+Think approach, setting a new standard for efficiency and performance in this space.
Reducing the capacity of large language models disproportionately impacts visual capabilities in multimodal systems, but visual extraction tuning combined with step-by-step reasoning improves efficiency and performance.
multimodal models large language model (LLM) downscaling visual capabilities visual extraction tuning Extract+Think approach