Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.
text
Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.
new_line
new_line
And now, everything is crumbling!
text
And now, everything is crumbling!
new_line
new_line
The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.
text
The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.
new_line
new_line
The model card in the Transformers behaves as Llama 3 and not 3.1.
text
The model card in the Transformers behaves as Llama 3 and not 3.1.
new_line
new_line
While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!
text
While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!
new_line
new_line
And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!
text
And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!
new_line
new_line
But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.
text
But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.
new_line
new_line
So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!
text
So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!
new_line
new_line
And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.
text
And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.
new_line
new_line
I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B
text
I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B
new_line
new_line
I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
text
I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
new_line
- "Think Before You Speak: Training Language Models With Pause Tokens"
text
- "Think Before You Speak: Training Language Models With Pause Tokens"
new_line
- "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"
text
- "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"
new_line
new_line
PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!
text
PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!
new_line
new_line
Great YT video: 
text
Great YT video: 
https://youtu.be/Xtr_Ll_A9ms
https://youtu.be/Xtr_Ll_A9ms
link
new_line
new_line
Hugging Face Clem Delangue 🤗?
text
Hugging Face Clem Delangue 🤗?
new_line
Can you please help here if possible? This will be the pinnacle of open-source!
text
Can you please help here if possible? This will be the pinnacle of open-source!
Reflection-Llama-3.1-70B burst onto the scene, surprising everyone! It claimed to outperform others with its novel Reflection-Tuning technique, promising not just to match but to surpass the likes of Claude 3.5 and GPT-4o, leveraging its 70 billion parameters to redefine what open-source could achieve.

And now, everything is crumbling!

The model's performance metrics, especially its 99.2% accuracy on the high school math dataset GSM 8K, have raised eyebrows. While it looked like a valedictorian, based on the open weights, it hardly performs like one.

The model card in the Transformers behaves as Llama 3 and not 3.1.

While the weights were released publicly, they are having issues aligning with the claims. The tuning has been restarted, and the author claims to upload the updated weights soon!

And the big one: the black-boxed API shared is not at all like the open weights. Even more, when pushed hard, the API endpoint claims to be an LLM by Anthropic!

But you might ask, didn't this model beat Anthropic Claude 3.5? Yes, it did.

So, did Claude 3.5 beat Claude 3.5? No, the benchmark is zero-shot, and the claims are that the results are not under zero-shot but under CoT/few-shot!

And to top it all off, the reflecting back idea is not new. But I don't think that's a big deal.

I took some time to look through everything, and now, once tested, this model looks to be worse than Llama 3.1 70B

I still believe the Reflection-Tuning technique is promising. These are the papers discussing its efficacy:
- "Think Before You Speak: Training Language Models With Pause Tokens"
- "Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning"

PS: Matt Shumer/@mattshumer_ (Twitter Handle) (Reflection-Llama-3.1-70B creator) is a great researcher. Let's wait for his updated weights!

Great YT video: https://youtu.be/Xtr_Ll_A9ms

Hugging Face Clem Delangue 🤗?
Can you please help here if possible? This will be the pinnacle of open-source!
👀
John6666 comarproject danielus louisbrulenaudet
😔
Forbu14
64b795b275e4fe20e9fc5aca
https://cdn-avatars.huggingface.co/v1/production/uploads/64b795b275e4fe20e9fc5aca/v5AWl4XUNF54_0hyRXqBc.jpeg
Ayaan Sharif
Ayaan-Sharif
user