Full fine-tuning of Microsoft's Phi2 on a single 4090 is now supported in axolotl. Thanks to 
text
Full fine-tuning of Microsoft's Phi2 on a single 4090 is now supported in axolotl. Thanks to 
@abacaj
mention
abacaj
 and 
text
 and 
@vikhyatk
mention
vikhyatk
 for their help with gradient checkpointing and flash attention fixes. 
text
 for their help with gradient checkpointing and flash attention fixes. 
new_line
new_line
alpaca finetune: 
text
alpaca finetune: 
https://huggingface.co/openaccess-ai-collective/phi2-alpaca
openaccess-ai-collective/phi2-alpaca
model
resource
https://huggingface.co/openaccess-ai-collective/phi2-alpaca
new_line
wandb: 
text
wandb: 
https://wandb.ai/oaaic/phi2/runs/00pc4ugb?workspace=user-wing-lian
https://wandb.ai/oaaic/phi2/runs/00pc4ugb?workspace=user-wing-lian
link
new_line
merged PR: 
text
merged PR: 
https://github.com/OpenAccess-AI-Collective/axolotl/pull/1058
https://github.com/OpenAccess-AI-Collective/axolotl/pull/1058
link
new_line
Full fine-tuning of Microsoft's Phi2 on a single 4090 is now supported in axolotl. Thanks to @abacaj and @vikhyatk for their help with gradient checkpointing and flash attention fixes. 

alpaca finetune: https://huggingface.co/openaccess-ai-collective/phi2-alpaca
wandb: https://wandb.ai/oaaic/phi2/runs/00pc4ugb?workspace=user-wing-lian
merged PR: https://github.com/OpenAccess-AI-Collective/axolotl/pull/1058

👍
osanseviero Citaman victor kramp Tonic marcsun13 merve ybelkada Zmu samusenps mrfakename Chunte JaiSurya radames attashe
❤️
osanseviero Tonic merve ybelkada Chunte radames SicariusSicariiStuff sbrandeis
5fcfb7c407408029ba3577e2
https://cdn-avatars.huggingface.co/v1/production/uploads/1608146735109-5fcfb7c407408029ba3577e2.png
Simon Brandeis
sbrandeis
user
641dfddf3bae5a77636817c5
https://cdn-avatars.huggingface.co/v1/production/uploads/641dfddf3bae5a77636817c5/2IwNwh9kK98eCHUmOGoWD.png
wing lian
winglian
user
62e54f0eae9d3f10acb95cb9
https://cdn-avatars.huggingface.co/v1/production/uploads/62e54f0eae9d3f10acb95cb9/VAyk05hqB3OZWXEZW-B0q.png
mrfakename
mrfakename
user
6538119803519fddb4a17e10
https://cdn-avatars.huggingface.co/v1/production/uploads/6538119803519fddb4a17e10/ffJMkdx-rM7VvLTCM6ri_.jpeg
samusenps
samusenps
user
6331f59718711776b46afb5e
/avatars/f18351bc5ce9c106ba74523d9a55567c.svg
Lone Striker
LoneStriker
user
638a3a2efe3185ae7324f1f6
https://cdn-avatars.huggingface.co/v1/production/uploads/1670003187019-noauth.png
brucethemoose
brucethemoose
user
6569216f9c96f1a47bf45788
https://cdn-avatars.huggingface.co/v1/production/uploads/6569216f9c96f1a47bf45788/mCLqmAs4dOjKdxNQVAp1w.png
Sica Rius
SicariusSicariiStuff
user