🎉 A new LLM is launched! 🚀 
text
🎉 A new LLM is launched! 🚀 
new_line
After checking if it's open-source or not, 🤔 
text
After checking if it's open-source or not, 🤔 
new_line
you rush to see the benchmarks... 🏃‍♂️💨
text
you rush to see the benchmarks... 🏃‍♂️💨
new_line
new_line
Which benchmark does everyone check first? 🔍
text
Which benchmark does everyone check first? 🔍
new_line
new_line
MMLU (Massive Multitask Language Understanding)? 📚
text
MMLU (Massive Multitask Language Understanding)? 📚
new_line
new_line
Benchmarks like MMLU reaching saturation... most of the time the performance does not translate to real-world use cases! 🌐❗
text
Benchmarks like MMLU reaching saturation... most of the time the performance does not translate to real-world use cases! 🌐❗
new_line
new_line
Meet MMLU-Pro, released by TIGER-Lab on 
text
Meet MMLU-Pro, released by TIGER-Lab on 
@huggingface
mention
huggingface
 ! 🐯🌍
text
 ! 🐯🌍
new_line
new_line
🧪 12,217 questions across biology, business, chemistry, computer science, economics, engineering, health, history, law, mathematics, philosophy, physics, and psychology carefully validated by humans 🧑‍🔬
text
🧪 12,217 questions across biology, business, chemistry, computer science, economics, engineering, health, history, law, mathematics, philosophy, physics, and psychology carefully validated by humans 🧑‍🔬
new_line
new_line
🔟 Goes to 10 options per question instead of 4, this increase in options will make the evaluation more realistic and reduce random guessing 🎯
text
🔟 Goes to 10 options per question instead of 4, this increase in options will make the evaluation more realistic and reduce random guessing 🎯
new_line
new_line
📊 56% of questions come from MMLU, 34% from STEM websites, and the rest from TheoremQA and SciBench 📈
text
📊 56% of questions come from MMLU, 34% from STEM websites, and the rest from TheoremQA and SciBench 📈
new_line
new_line
🤖 LLMs with weak chain-of-thought reasoning tend to perform lower, indicating it is more challenging and representative of real-world expectations 🧠💡
text
🤖 LLMs with weak chain-of-thought reasoning tend to perform lower, indicating it is more challenging and representative of real-world expectations 🧠💡
new_line
new_line
Any guess who tops it and who bombs it? 🤔📉📈
text
Any guess who tops it and who bombs it? 🤔📉📈
new_line
new_line
GPT-4o drops by 17% (from 0.887 to 0.7149) 📉
text
GPT-4o drops by 17% (from 0.887 to 0.7149) 📉
new_line
Llama-3-70B drops by 27% (from 0.820 to 0.5541) 📉
text
Llama-3-70B drops by 27% (from 0.820 to 0.5541) 📉
new_line
new_line
🔗 
text
🔗 
https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
TIGER-Lab/MMLU-Pro
dataset
resource
https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
🎉 A new LLM is launched! 🚀 
After checking if it's open-source or not, 🤔 
you rush to see the benchmarks... 🏃‍♂️💨

Which benchmark does everyone check first? 🔍

MMLU (Massive Multitask Language Understanding)? 📚

Benchmarks like MMLU reaching saturation... most of the time the performance does not translate to real-world use cases! 🌐❗

Meet MMLU-Pro, released by TIGER-Lab on @huggingface ! 🐯🌍

🧪 12,217 questions across biology, business, chemistry, computer science, economics, engineering, health, history, law, mathematics, philosophy, physics, and psychology carefully validated by humans 🧑‍🔬

🔟 Goes to 10 options per question instead of 4, this increase in options will make the evaluation more realistic and reduce random guessing 🎯

📊 56% of questions come from MMLU, 34% from STEM websites, and the rest from TheoremQA and SciBench 📈

🤖 LLMs with weak chain-of-thought reasoning tend to perform lower, indicating it is more challenging and representative of real-world expectations 🧠💡

Any guess who tops it and who bombs it? 🤔📉📈

GPT-4o drops by 17% (from 0.887 to 0.7149) 📉
Llama-3-70B drops by 27% (from 0.820 to 0.5541) 📉

🔗 https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
🔥
clem zaursamedov1 osanseviero
❤️
clem zaursamedov1 clefourrier
5e67bdd61009063689407479
https://cdn-avatars.huggingface.co/v1/production/uploads/1583857146757-5e67bdd61009063689407479.jpeg
Clem 🤗
clem
user
662c66989e6d371ab7b7f414
/avatars/e4fa0d942a2e23bffb365601a34c4c9d.svg
Z
Zachyyypoo
user