A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
The surge of interest towards Multi-modal Large Language Models (MLLMs),
e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both
academia and industry. They endow Large Language Models (LLMs) with powerful
capabilities in visual understanding, enabling them to tackle diverse
multi-modal tasks. Very recently, Google released Gemini, its newest and most
capable MLLM built from the ground up for multi-modality. In light of the
superior reasoning capabilities, can Gemini challenge GPT-4V's leading position
in multi-modal learning? In this paper, we present a preliminary exploration of
Gemini Pro's visual understanding proficiency, which comprehensively covers
four domains: fundamental perception, advanced cognition, challenging vision
tasks, and various expert capacities. We compare Gemini Pro with the
state-of-the-art GPT-4V to evaluate its upper limits, along with the latest
open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and
black-box systems. The qualitative samples indicate that, while GPT-4V and
Gemini showcase different answering styles and preferences, they can exhibit
comparable visual reasoning capabilities, and Sphinx still trails behind them
concerning domain generalizability. Specifically, GPT-4V tends to elaborate
detailed explanations and intermediate steps, and Gemini prefers to output a
direct and concise answer. The quantitative evaluation on the popular MME
benchmark also demonstrates the potential of Gemini to be a strong challenger
to GPT-4V. Our early investigation of Gemini also observes some common issues
of MLLMs, indicating that there still remains a considerable distance towards
artificial general intelligence. Our project for tracking the progress of MLLM
is released at
https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.
6582505d1c4454dde6d3a6aa
Chaoyou Fu
admin_assigned
2023-12-20T09:12:04.184Z
653b268cd1041ca9188954da
https://cdn-avatars.huggingface.co/v1/production/uploads/653b268cd1041ca9188954da/5Qy0fyQdRCIUUAzV6fg1k.png
Chaoyou Fu
BradyFU
user
BradyFU
6582505d1c4454dde6d3a6ab
Renrui Zhang
claimed_verified
2024-03-22T08:40:05.948Z
645b8b2687c79b6ec0bb3b7a
/avatars/00a9db32a42dc950112bf2593bb109cb.svg
Renrui
ZrrSkywalker
user
ZrrSkywalker
6582505d1c4454dde6d3a6ac
Haojia Lin
admin_assigned
2023-12-20T09:13:21.749Z
64ffd436d522560505a94b8e
/avatars/02d4faac40ac203cb5d635cfcb39780c.svg
Haojia Lin
linhaojia13
user
linhaojia13
6582505d1c4454dde6d3a6ad
Zihan Wang
6582505d1c4454dde6d3a6ae
Timin Gao
6582505d1c4454dde6d3a6af
Yongdong Luo
6582505d1c4454dde6d3a6b0
Yubo Huang
6582505d1c4454dde6d3a6b1
Zhengye Zhang
6582505d1c4454dde6d3a6b2
Longtian Qiu
6582505d1c4454dde6d3a6b3
Gaoxiang Ye
6582505d1c4454dde6d3a6b4
Yunhang Shen
admin_assigned
2023-12-20T09:19:11.278Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
shenyunhang
user
shenyunhang
6582505d1c4454dde6d3a6b5
Mengdan Zhang
6582505d1c4454dde6d3a6b6
Peixian Chen
6582505d1c4454dde6d3a6b7
Sirui Zhao
admin_assigned
2023-12-20T09:18:50.406Z
653a55ddae6a4080ce3eb860
/avatars/44a62d1e0b72a42690d1658e5b42f3a4.svg
Sirui Zhao
SiruiZhao
user
SiruiZhao
6582505d1c4454dde6d3a6b8
Xiawu Zheng
6582505d1c4454dde6d3a6b9
Shaohui Lin
6582505d1c4454dde6d3a6ba
Deqiang Jiang
6582505d1c4454dde6d3a6bb
Di Yin
6582505d1c4454dde6d3a6bc
Peng Gao
6582505d1c4454dde6d3a6bd
Ke Li
claimed_verified
2025-01-09T20:02:12.226Z
63280915eeee4dd858083092
/avatars/78347af4af42527d53e88d9969c5c934.svg
Ke Li
tristanli
user
tristanli
6582505d1c4454dde6d3a6be
Xing Sun
admin_assigned
2023-12-20T09:16:45.196Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
tedsun
user
tedsun
6582505d1c4454dde6d3a6bf
Rongrong Ji
A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
The surge of interest towards Multi-modal Large Language Models (MLLMs),
e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both
academia and industry. They endow Large Language Models (LLMs) with powerful
capabilities in visual understanding, enabling them to tackle diverse
multi-modal tasks. Very recently, Google released Gemini, its newest and most
capable MLLM built from the ground up for multi-modality. In light of the
superior reasoning capabilities, can Gemini challenge GPT-4V's leading position
in multi-modal learning? In this paper, we present a preliminary exploration of
Gemini Pro's visual understanding proficiency, which comprehensively covers
four domains: fundamental perception, advanced cognition, challenging vision
tasks, and various expert capacities. We compare Gemini Pro with the
state-of-the-art GPT-4V to evaluate its upper limits, along with the latest
open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and
black-box systems. The qualitative samples indicate that, while GPT-4V and
Gemini showcase different answering styles and preferences, they can exhibit
comparable visual reasoning capabilities, and Sphinx still trails behind them
concerning domain generalizability. Specifically, GPT-4V tends to elaborate
detailed explanations and intermediate steps, and Gemini prefers to output a
direct and concise answer. The quantitative evaluation on the popular MME
benchmark also demonstrates the potential of Gemini to be a strong challenger
to GPT-4V. Our early investigation of Gemini also observes some common issues
of MLLMs, indicating that there still remains a considerable distance towards
artificial general intelligence. Our project for tracking the progress of MLLM
is released at
https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.
Preliminary evaluation of Gemini Pro's visual understanding shows it to be a strong contender to GPT-4V in multi-modal learning, though both models face challenges in achieving domain generalization compared to open-sourced systems.
Multi-modal Large Language Models MLLMs GPT-4V Gemini visual understanding fundamental perception advanced cognition challenging vision tasks expert capacities MME benchmark domain generalization artificial general intelligence