𝗧𝗵𝗲 𝗵𝘂𝗴𝗲 𝗰𝗼𝘀𝘁 𝗼𝗳 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗼𝗻 𝗳𝗿𝗼𝗻𝘁𝗶𝗲𝗿 𝗟𝗟𝗠𝘀 💸
text
𝗧𝗵𝗲 𝗵𝘂𝗴𝗲 𝗰𝗼𝘀𝘁 𝗼𝗳 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗼𝗻 𝗳𝗿𝗼𝗻𝘁𝗶𝗲𝗿 𝗟𝗟𝗠𝘀 💸
new_line
new_line
Google DeepMind recently released a great paper that shows optimal hyperparameters to train across different regimes: Scaling Exponents Across Parameterizations and Optimizers, with data from 10,000 training runs.
text
Google DeepMind recently released a great paper that shows optimal hyperparameters to train across different regimes: Scaling Exponents Across Parameterizations and Optimizers, with data from 10,000 training runs.
new_line
new_line
One engineer decided to quantify the price of such a large-scale experiment.
text
One engineer decided to quantify the price of such a large-scale experiment.
new_line
new_line
😬 And the bill is hefty: ~13M USD 
text
😬 And the bill is hefty: ~13M USD 
new_line
new_line
This exact number is to take with a grain of salt because many approximations were necessary to get the final result.
text
This exact number is to take with a grain of salt because many approximations were necessary to get the final result.
new_line
new_line
⛔️ But still this ballpark means that for this sole experiment, the price is way over what most startups or research labs could afford.
text
⛔️ But still this ballpark means that for this sole experiment, the price is way over what most startups or research labs could afford.
new_line
new_line
This means that open-sourcing research is more important than ever, to put everyone in the ecosystem on a roughly equal footing. Don't let OpenAI run first, they'll keep everything for themselves!
text
This means that open-sourcing research is more important than ever, to put everyone in the ecosystem on a roughly equal footing. Don't let OpenAI run first, they'll keep everything for themselves!
new_line
new_line
Read the full post that quantifies the paper's cost 👉 
text
Read the full post that quantifies the paper's cost 👉 
https://152334h.github.io/blog/scaling-exponents/
https://152334h.github.io/blog/scaling-exponents/
link
𝗧𝗵𝗲 𝗵𝘂𝗴𝗲 𝗰𝗼𝘀𝘁 𝗼𝗳 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗼𝗻 𝗳𝗿𝗼𝗻𝘁𝗶𝗲𝗿 𝗟𝗟𝗠𝘀 💸

Google DeepMind recently released a great paper that shows optimal hyperparameters to train across different regimes: Scaling Exponents Across Parameterizations and Optimizers, with data from 10,000 training runs.

One engineer decided to quantify the price of such a large-scale experiment.

😬 And the bill is hefty: ~13M USD 

This exact number is to take with a grain of salt because many approximations were necessary to get the final result.

⛔️ But still this ballpark means that for this sole experiment, the price is way over what most startups or research labs could afford.

This means that open-sourcing research is more important than ever, to put everyone in the ecosystem on a roughly equal footing. Don't let OpenAI run first, they'll keep everything for themselves!

Read the full post that quantifies the paper's cost 👉 https://152334h.github.io/blog/scaling-exponents/
👀
nbroad MicPie louisbrulenaudet adamo1139 victor
🔥
pduf
66a8985e5bd5c24dec21f0bd
/avatars/7411dc98c012221f8e4c3641c8702640.svg
juliawhites
juliawhites
user