~75% on the challenging GPQA with only 40M parameters 🔥🥳
text
~75% on the challenging GPQA with only 40M parameters 🔥🥳
new_line
new_line
GREAT ACHIEVEMENT ! Or is it ?
text
GREAT ACHIEVEMENT ! Or is it ?
new_line
new_line
This new Work, "Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation", take out the mystery about many models i personally suspected their results. Speacially on leaderboards other than the english one, Like the Open Arabic LLM Leaderbaord 
text
This new Work, "Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation", take out the mystery about many models i personally suspected their results. Speacially on leaderboards other than the english one, Like the Open Arabic LLM Leaderbaord 
https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard
OALL/Open-Arabic-LLM-Leaderboard
space
resource
https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard
.
text
.
new_line
new_line
The authors of this work, first started by training a model on the GPQA data, which, unsurprisingly, led to the model achieving 100% performance.
text
The authors of this work, first started by training a model on the GPQA data, which, unsurprisingly, led to the model achieving 100% performance.
new_line
new_line
Afterward, they trained what they referred to as a 'legitimate' model on legitimate data (MedMCQA). However, they introduced a distillation loss from the earlier, 'cheated' model.
text
Afterward, they trained what they referred to as a 'legitimate' model on legitimate data (MedMCQA). However, they introduced a distillation loss from the earlier, 'cheated' model.
new_line
new_line
What they discovered was fascinating: the knowledge of GPQA leaked through this distillation loss, even though the legitimate model was never explicitly trained on GPQA during this stage.
text
What they discovered was fascinating: the knowledge of GPQA leaked through this distillation loss, even though the legitimate model was never explicitly trained on GPQA during this stage.
new_line
new_line
This raises important questions about the careful use of distillation in model training, especially when the training data is opaque. As they demonstrated, it’s apparently possible to (intentionally or unintentionally) leak test data through this method.
text
This raises important questions about the careful use of distillation in model training, especially when the training data is opaque. As they demonstrated, it’s apparently possible to (intentionally or unintentionally) leak test data through this method.
new_line
new_line
Find out more: 
text
Find out more: 
Data Laundering: Artificially Boosting Benchmark Results through
  Knowledge Distillation (2412.15255)
https://huggingface.co/papers/2412.15255
2412.15255
paper
resource
https://huggingface.co/papers/2412.15255
~75% on the challenging GPQA with only 40M parameters 🔥🥳

GREAT ACHIEVEMENT ! Or is it ?

This new Work, "Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation", take out the mystery about many models i personally suspected their results. Speacially on leaderboards other than the english one, Like the Open Arabic LLM Leaderbaord https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard.

The authors of this work, first started by training a model on the GPQA data, which, unsurprisingly, led to the model achieving 100% performance.

Afterward, they trained what they referred to as a 'legitimate' model on legitimate data (MedMCQA). However, they introduced a distillation loss from the earlier, 'cheated' model.

What they discovered was fascinating: the knowledge of GPQA leaked through this distillation loss, even though the legitimate model was never explicitly trained on GPQA during this stage.

This raises important questions about the careful use of distillation in model training, especially when the training data is opaque. As they demonstrated, it’s apparently possible to (intentionally or unintentionally) leak test data through this method.

Find out more: https://huggingface.co/papers/2412.15255
🔥
John6666 djuna reonyy AtAndDev
676a66e3b9c4140ab3a6d6a4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yV4KiM5_G-GPtvaAcOQlu.png
devass
devass121
user