Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL
69ae352b9893f58272f740e2
Boqiang Zhang
claimed_verified
2026-03-09T08:47:30.867Z
626938b16f8f86ad21deb989
/avatars/b577835cb6053c90c1e86b7376c6104b.svg
Boqiang Zhang
Cyril666
user
Cyril666
69ae352b9893f58272f740e3
Lei Ke
claimed_verified
2026-03-09T08:46:58.434Z
6258a6455ea3a0a9b6de3f22
/avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
Lei Ke
lkeab
user
lkeab
69ae352b9893f58272f740e4
Ruihan Yang
claimed_verified
2026-03-09T08:47:33.365Z
64acd8872d29d3bdcd422332
/avatars/922cb295599aec16b90233bfd2f14524.svg
Ruihan Yang
rhyang
user
rhyang
69ae352b9893f58272f740e5
Qi Gao
claimed_verified
2026-03-09T13:50:45.995Z
663d9a5bfb712ba4f35cf847
/avatars/c6957cb5ef17b3da8e2130e530926a84.svg
Qi Gao
qgao21
user
qgao21
69ae352b9893f58272f740e6
Tianyuan Qu
claimed_verified
2026-03-09T08:56:06.742Z
66e79b3c1c79fc2e51dc1d60
/avatars/8706336e9e7a417505c9bb32583a662f.svg
QU Tianyuan
TainU
user
TainU
69ae352b9893f58272f740e7
Rossell Chen
69ae352b9893f58272f740e8
Dong Yu
69ae352b9893f58272f740e9
Leoweiliang
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: https://github.com/tencent-ailab/Penguin-VL
Penguin-VL demonstrates that text-only initialized vision encoders can achieve superior performance in multimodal understanding tasks compared to traditional contrastive pretraining methods, enabling efficient deployment on resource-constrained devices.
Vision Language Model contrastive learning vision encoder text-only LLM multimodal understanding dense captioning visual reasoning computational efficiency model scaling visual fidelity