I've arrived at an interesting result on the current Open LLM leaderboard.
text
I've arrived at an interesting result on the current Open LLM leaderboard.
new_line
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/
open-llm-leaderboard/open_llm_leaderboard
space
resource
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/
new_line
After I narrowed down the filter of models to be between 8-9B parameters, my recent merge of o1 reasoning models achieved the highest MATH eval result of any Llama 3.x 8B model currently on the board, hitting 33.99%, placing 973/2795.
text
After I narrowed down the filter of models to be between 8-9B parameters, my recent merge of o1 reasoning models achieved the highest MATH eval result of any Llama 3.x 8B model currently on the board, hitting 33.99%, placing 973/2795.
new_line
https://huggingface.co/grimjim/HuatuoSkywork-o1-Llama-3.1-8B
grimjim/HuatuoSkywork-o1-Llama-3.1-8B
model
resource
https://huggingface.co/grimjim/HuatuoSkywork-o1-Llama-3.1-8B
new_line
new_line
Unfortunately, I need more information to evaluate the parent models used in the merge.
text
Unfortunately, I need more information to evaluate the parent models used in the merge.
new_line
The Skywork/Skywork-o1-Open-Llama-3.1-8B model scored 0% on the MATH eval, which I suspect was due to output formatting that was baked too hard into the model, and placed 2168/2795; the merge achieved a significant uplift in every benchmark across the board.
text
The Skywork/Skywork-o1-Open-Llama-3.1-8B model scored 0% on the MATH eval, which I suspect was due to output formatting that was baked too hard into the model, and placed 2168/2795; the merge achieved a significant uplift in every benchmark across the board.
new_line
Unfortunately, FreedomIntelligence/HuatuoGPT-o1-8B was not currently benched as of this post, so I am unable to assess relative benchmarks. Nevertheless, it is intriguing that an ostensibly medical o1 model appears to have resulted in a sizable MATH boost.
text
Unfortunately, FreedomIntelligence/HuatuoGPT-o1-8B was not currently benched as of this post, so I am unable to assess relative benchmarks. Nevertheless, it is intriguing that an ostensibly medical o1 model appears to have resulted in a sizable MATH boost.
I've arrived at an interesting result on the current Open LLM leaderboard.
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/
After I narrowed down the filter of models to be between 8-9B parameters, my recent merge of o1 reasoning models achieved the highest MATH eval result of any Llama 3.x 8B model currently on the board, hitting 33.99%, placing 973/2795.
https://huggingface.co/grimjim/HuatuoSkywork-o1-Llama-3.1-8B

Unfortunately, I need more information to evaluate the parent models used in the merge.
The Skywork/Skywork-o1-Open-Llama-3.1-8B model scored 0% on the MATH eval, which I suspect was due to output formatting that was baked too hard into the model, and placed 2168/2795; the merge achieved a significant uplift in every benchmark across the board.
Unfortunately, FreedomIntelligence/HuatuoGPT-o1-8B was not currently benched as of this post, so I am unable to assess relative benchmarks. Nevertheless, it is intriguing that an ostensibly medical o1 model appears to have resulted in a sizable MATH boost.
👍
John6666 AtAndDev
🔥
darkzbaron AtAndDev