🚀 **InfiniTransformer, Gemma/Llama3 based Implementation!** 🌌
text
🚀 **InfiniTransformer, Gemma/Llama3 based Implementation!** 🌌
new_line
new_line
> Update @ 2024.04.19: It now supports Llama-3!
text
> Update @ 2024.04.19: It now supports Llama-3!
new_line
new_line
> Note: this implementation is unofficial
text
> Note: this implementation is unofficial
new_line
new_line
This implementation is designed to handle virtually infinite context lengths.
text
This implementation is designed to handle virtually infinite context lengths.
new_line
new_line
Here's the github repo: 
text
Here's the github repo: 
https://github.com/Beomi/InfiniTransformer
https://github.com/Beomi/InfiniTransformer
link
new_line
new_line
📄 **Read the original Paper:** 
text
📄 **Read the original Paper:** 
https://arxiv.org/abs/2404.07143
https://arxiv.org/abs/2404.07143
link
new_line
new_line
## **Focus on Infini-Attention**
text
## **Focus on Infini-Attention**
new_line
new_line
- **2 Types of Implementation available:** Attention-layer only implementation / Model & Train-wise implementation
text
- **2 Types of Implementation available:** Attention-layer only implementation / Model & Train-wise implementation
new_line
- **Fixed(segment dependent) Memory Usage:** Enables training on larger models and longer sequences without the memory overhead typical of standard Transformer implementations.
text
- **Fixed(segment dependent) Memory Usage:** Enables training on larger models and longer sequences without the memory overhead typical of standard Transformer implementations.
new_line
- **Infinite Context Capability:** Train with unprecedented sequence lengths—imagine handling up to 1 million sequence lengths on standard hardware!
text
- **Infinite Context Capability:** Train with unprecedented sequence lengths—imagine handling up to 1 million sequence lengths on standard hardware!
new_line
  - You could train Gemma-2B with 1M sequence length with 2K segmentation size with single H100 GPU.
text
  - You could train Gemma-2B with 1M sequence length with 2K segmentation size with single H100 GPU.
new_line
new_line
## **Try InfiniTransformer**
text
## **Try InfiniTransformer**
new_line
new_line
1. **Clone the repository:**
text
1. **Clone the repository:**
new_line
text
``
inline_code
bash
   git clone https://github.com/Beomi/InfiniTransformer
   
`bash
   git clone https://github.com/Beomi/InfiniTransformer
   `
inline_code
``
inline_code
new_line
2. **Install necessary tools:**
text
2. **Install necessary tools:**
new_line
text
``
inline_code
bash
   pip install -r requirements.txt
   pip install -e git+https://github.com/huggingface/transformers.git@b109257f4f#egg=transformers
   
`bash
   pip install -r requirements.txt
   pip install -e git+https://github.com/huggingface/transformers.git@b109257f4f#egg=transformers
   `
inline_code
``
inline_code
new_line
3. **Dive Deep into Custom Training:**
text
3. **Dive Deep into Custom Training:**
new_line
   - Train with extensive sequence lengths using scripts such as 
text
   - Train with extensive sequence lengths using scripts such as 
./train.gemma.infini.noclm.1Mseq.sh
`./train.gemma.infini.noclm.1Mseq.sh`
inline_code
.
text
.
new_line
new_line
for more detailed info, please visit Repo: 
text
for more detailed info, please visit Repo: 
https://github.com/Beomi/InfiniTransformer
https://github.com/Beomi/InfiniTransformer
link
new_line
new_line
Look forward to see your feedbacks! 😊
text
Look forward to see your feedbacks! 😊
new_line
new_line
ps. Training loss plot is here 😉
text
ps. Training loss plot is here 😉
🚀 **InfiniTransformer, Gemma/Llama3 based Implementation!** 🌌

> Update @ 2024.04.19: It now supports Llama-3!

> Note: this implementation is unofficial

This implementation is designed to handle virtually infinite context lengths.

Here's the github repo: https://github.com/Beomi/InfiniTransformer

📄 **Read the original Paper:** https://arxiv.org/abs/2404.07143

## **Focus on Infini-Attention**

- **2 Types of Implementation available:** Attention-layer only implementation / Model & Train-wise implementation
- **Fixed(segment dependent) Memory Usage:** Enables training on larger models and longer sequences without the memory overhead typical of standard Transformer implementations.
- **Infinite Context Capability:** Train with unprecedented sequence lengths—imagine handling up to 1 million sequence lengths on standard hardware!
  - You could train Gemma-2B with 1M sequence length with 2K segmentation size with single H100 GPU.

## **Try InfiniTransformer**

1. **Clone the repository:**
   ```bash
   git clone https://github.com/Beomi/InfiniTransformer
   ```
2. **Install necessary tools:**
   ```bash
   pip install -r requirements.txt
   pip install -e git+https://github.com/huggingface/transformers.git@b109257f4f#egg=transformers
   ```
3. **Dive Deep into Custom Training:**
   - Train with extensive sequence lengths using scripts such as `./train.gemma.infini.noclm.1Mseq.sh`.

for more detailed info, please visit Repo: https://github.com/Beomi/InfiniTransformer

Look forward to see your feedbacks! 😊

ps. Training loss plot is here 😉
🚀
tomaarsen maywell adamm-hf hllj Joseph717171 genne HyeonjinXZ
🔥
flozi00 raidhon Joseph717171 sosoai
👍
gangyeolkim Joseph717171 nebchi
6317233cc92fd6fee317e030
https://cdn-avatars.huggingface.co/v1/production/uploads/6317233cc92fd6fee317e030/cJHSvvimr1kqgQfHOjO5n.png
Tom Aarsen
tomaarsen
user
5e56829137cb5b49818287ea
https://cdn-avatars.huggingface.co/v1/production/uploads/5e56829137cb5b49818287ea/8HYzJeRc4b9Wu7BfJwibS.png
Lee Junbum
beomi
user