Remember Gemini, GPT-4o, all being true multimodal models 🌟.
text
Remember Gemini, GPT-4o, all being true multimodal models 🌟.
new_line
new_line
Now we have a paper 📄 describing an architecture that might achieve that!
text
Now we have a paper 📄 describing an architecture that might achieve that!
new_line
new_line
Uni-MoE: a native multimodal, Unified Mixture of Experts (MoE) architecture 🏗️.
text
Uni-MoE: a native multimodal, Unified Mixture of Experts (MoE) architecture 🏗️.
new_line
new_line
Uni-MoE integrates various modalities (text 📝, image 🖼️, audio 🎵, video 📹, speech 🗣️) using modality-specific encoders and connectors for a cohesive multimodal understanding.
text
Uni-MoE integrates various modalities (text 📝, image 🖼️, audio 🎵, video 📹, speech 🗣️) using modality-specific encoders and connectors for a cohesive multimodal understanding.
new_line
new_line
Training Strategy:
text
Training Strategy:
new_line
1️⃣ Training cross-modality alignment with diverse connectors 🔄.
text
1️⃣ Training cross-modality alignment with diverse connectors 🔄.
new_line
2️⃣ Training modality-specific experts using cross-modality instruction data 📊.
text
2️⃣ Training modality-specific experts using cross-modality instruction data 📊.
new_line
3️⃣Tuning the Uni-MoE framework with Low-Rank Adaptation (LoRA) on mixed multimodal data 🔧.
text
3️⃣Tuning the Uni-MoE framework with Low-Rank Adaptation (LoRA) on mixed multimodal data 🔧.
new_line
new_line
Technical Details:
text
Technical Details:
new_line
new_line
Modality-Specific Encoders: CLIP for images 🖼️, Whisper for speech 🗣️, BEATs for audio 🎵.
text
Modality-Specific Encoders: CLIP for images 🖼️, Whisper for speech 🗣️, BEATs for audio 🎵.
new_line
new_line
MoE-Based Blocks: Shared self-attention layers, feed-forward networks (FFN) based experts, and sparse routers for token-level expertise allocation 🚀.
text
MoE-Based Blocks: Shared self-attention layers, feed-forward networks (FFN) based experts, and sparse routers for token-level expertise allocation 🚀.
new_line
new_line
Efficient Training: Utilizes LoRA for fine-tuning pre-trained experts and self-attention layers 🛠️.
text
Efficient Training: Utilizes LoRA for fine-tuning pre-trained experts and self-attention layers 🛠️.
new_line
new_line
Uni-MoE outperforms traditional dense models on benchmarks like A-OKVQA, OK-VQA, VQAv2, MMBench, RACE-Audio, and English High School Listening Test 🏆.
text
Uni-MoE outperforms traditional dense models on benchmarks like A-OKVQA, OK-VQA, VQAv2, MMBench, RACE-Audio, and English High School Listening Test 🏆.
new_line
new_line
The code is open-sourced as well: 
text
The code is open-sourced as well: 
https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs/tree/master/Uni_MoE_v2
https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs/tree/master/Uni_MoE_v2
link
new_line
new_line
Paper: 
text
Paper: 
Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts (2405.11273)
https://huggingface.co/papers/2405.11273
2405.11273
paper
resource
https://huggingface.co/papers/2405.11273
Remember Gemini, GPT-4o, all being true multimodal models 🌟.

Now we have a paper 📄 describing an architecture that might achieve that!

Uni-MoE: a native multimodal, Unified Mixture of Experts (MoE) architecture 🏗️.

Uni-MoE integrates various modalities (text 📝, image 🖼️, audio 🎵, video 📹, speech 🗣️) using modality-specific encoders and connectors for a cohesive multimodal understanding.

Training Strategy:
1️⃣ Training cross-modality alignment with diverse connectors 🔄.
2️⃣ Training modality-specific experts using cross-modality instruction data 📊.
3️⃣Tuning the Uni-MoE framework with Low-Rank Adaptation (LoRA) on mixed multimodal data 🔧.

Technical Details:

Modality-Specific Encoders: CLIP for images 🖼️, Whisper for speech 🗣️, BEATs for audio 🎵.

MoE-Based Blocks: Shared self-attention layers, feed-forward networks (FFN) based experts, and sparse routers for token-level expertise allocation 🚀.

Efficient Training: Utilizes LoRA for fine-tuning pre-trained experts and self-attention layers 🛠️.

Uni-MoE outperforms traditional dense models on benchmarks like A-OKVQA, OK-VQA, VQAv2, MMBench, RACE-Audio, and English High School Listening Test 🏆.

The code is open-sourced as well: https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs/tree/master/Uni_MoE_v2

Paper: https://huggingface.co/papers/2405.11273
🤗
Lumpen1 alielfilali01 gilangf3000
🚀
osanseviero alielfilali01
🧠
alielfilali01