Henyo-70M

Henyo is a custom Tagalog LLM trained on a subset of Wikipedia.

Model Architecture

• Parameter Count: 70M
• Architecture: Decoder-only Transformer (Custom)
• Features:
  • SwiGLU Activation
  • Grouped Query Attention (GQA)
  • Rotary Positional Embeddings (RoPE)
  • RMSNorm

Usage

This model uses a custom architecture code. You can load it using the AutoModel class with trust_remote_code=True (if code is uploaded) or by defining the class manually.

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("marcuscedricridia/Henyo-70M")
# Note: Since this is a custom model, you may need the inference script provided in the repo.