Instructions to use AuroraAI-Research/Aurora2-400M-A20M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AuroraAI-Research/Aurora2-400M-A20M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="AuroraAI-Research/Aurora2-400M-A20M", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("AuroraAI-Research/Aurora2-400M-A20M", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use AuroraAI-Research/Aurora2-400M-A20M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AuroraAI-Research/Aurora2-400M-A20M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AuroraAI-Research/Aurora2-400M-A20M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/AuroraAI-Research/Aurora2-400M-A20M
- SGLang
How to use AuroraAI-Research/Aurora2-400M-A20M with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AuroraAI-Research/Aurora2-400M-A20M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AuroraAI-Research/Aurora2-400M-A20M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AuroraAI-Research/Aurora2-400M-A20M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AuroraAI-Research/Aurora2-400M-A20M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use AuroraAI-Research/Aurora2-400M-A20M with Docker Model Runner:
docker model run hf.co/AuroraAI-Research/Aurora2-400M-A20M
Aurora2-400M-A20M
Aurora2-400M-A20M is a decoder-only language model with approximately 400M total parameters and 19M active parameters per token. It uses a top-1 Mixture-of-Experts (MoE) architecture with 32 routed experts and one shared expert per layer.
The model was pretrained from scratch on 100B tokens using a single GPU. It is a base model, not an instruction-tuned assistant: there was no supervised instruction-tuning or RLHF stage.
| Property | Value |
|---|---|
| Total parameters | 399,597,454 |
| Active parameters per token | 19,050,382 |
| Experts | 32 routed + 1 shared per layer |
| Layers | 12 |
| Hidden size | 192 |
| Context length | 2,048 tokens |
| Vocabulary | 16,384 |
| Training tokens | 100B |
| Training precision | BF16 compute, FP32 master weights |
| Hardware | Single RTX Pro 6000, approximately 36 hours |
Architecture
Aurora2 uses a relatively narrow attention stack paired with large, sparse feed-forward layers. Each token is routed to one of 32 experts, alongside the always-active shared expert.
Core configuration
| Component | Configuration |
|---|---|
| Hidden size | 192 |
| Layers | 12 |
| Attention heads | 6 query / 2 KV |
| Attention type | Grouped-query attention (3:1) |
| Head dimension | 32 |
| Positional encoding | RoPE, base 10,000 |
| Normalization | Pre-RMSNorm, FP32 QK normalization |
| Attention | FlexAttention, causal and per-document masking |
| Routed experts | 32 per layer, SwiGLU, FFN dimension 1,776 |
| Shared expert | 1 per layer, SwiGLU, FFN dimension 256 |
| Routing | FP32 softmax, top-1, dropless |
| Output head | Tied token embeddings |
| Logit soft-capping | 30 |
The router scales each selected expert's output by its routing probability. There is no expert capacity factor or token dropping.
Router training includes a Switch-style load-balancing loss (coefficient 0.01) and router z-loss (coefficient 0.001).
Additional components
- Hashed n-gram embeddings: Causal hashes of the last 2, 3 and 4 token IDs index three separate 16,384-bucket embedding tables. Their outputs are concatenated, projected to the hidden dimension and added to the token embeddings through a learned per-channel gate.
- Embedding skip connections: Every block combines its input with the original embedding using learned coefficients, initialized to
[1, 0]. - Value residuals: Layers after the first mix their value tensors with the first layer's values, using learned coefficients initialized to
[0.5, 0.5]. - Per-head attention gates: Each attention head has a learned output gate derived from the first 12 channels of the normalized block input. It is initialized to produce a multiplier of 1.
- Learned router bias: Each expert has a trainable bias added to its routing logits.
Tokenizer
Aurora2 uses a byte-level BPE tokenizer trained from scratch.
- Vocabulary size: 16,384.
- Every digit is an individual token (
"123"becomes three tokens). - Special tokens:
<|bos|>: ID 0<|eos|>: ID 1<|pad|>: ID 2
- The tokenizer automatically prepends
<|bos|>to encoded text.
During training, documents were packed together, with each document starting with a BOS token. No EOS was inserted between documents. BOS tokens also served as document boundaries for attention masking.
Training data
Training data was streamed from the Hugging Face Hub, tokenized on the fly and packed into sequences of 2,048 tokens. The tokenizer was trained on approximately 1B characters from the same data mixture.
Only training splits were used. ClimbMix shard 6542 was reserved for validation; benchmark evaluation splits were not used for training.
| Dataset | Purpose | Average token share | Approx. epochs |
|---|---|---|---|
| ClimbMix | General web text | 85.8% | < 1 |
| FineMath-4+ | Mathematical web text | 6.0% | < 1 |
| Wikipedia (English) | Encyclopedic text | 3.0% | < 1 |
| SmolTalk | Chat and QA | 2.5% | ~2.5 |
| OpenMathInstruct-2 | Mathematical QA | 1.5% | ~2 |
| NuminaMath-CoT | Mathematical QA | 1.2% | ~3 |
Staged data mixture
The mixture was adjusted throughout training. Math and QA/chat sources received greater weight during the final decay phase, while ClimbMix remained the majority source.
| Training stage | ClimbMix | FineMath | Wikipedia | SmolTalk | OpenMath | Numina |
|---|---|---|---|---|---|---|
| 0–60B tokens | 89.3% | 5.4% | 2.7% | 1.25% | 0.75% | 0.6% |
| 60–80B tokens | 85.8% | 6.0% | 3.0% | 2.5% | 1.5% | 1.2% |
| 80–100B tokens | 75.3% | 7.8% | 3.9% | 6.25% | 3.75% | 3.0% |
Instruction-style datasets were converted into plain-text formats, including Question: ... Answer: ..., Problem: ... Solution: ... and User: ... Assistant: .... These examples were part of pretraining; they do not constitute an instruction-tuning stage.
Training procedure
| Hyperparameter | Value |
|---|---|
| Total training tokens | 100B |
| Sequences per step | 128 |
| Sequence length | 2,048 |
| Tokens per step | 262,144 |
| Approximate optimizer steps | 381,000 |
| Learning-rate schedule | Warmup-Stable-Decay (WSD) |
| Warmup | 1B tokens, linear |
| Stable phase | 1B–80B tokens |
| Decay | Cosine decay over the final 20B tokens |
| Peak learning rate | 1.5e-3 |
| Router learning-rate multiplier | 0.5 |
| Optimizer | Muon + AdamW |
| Gradient clipping | 1.0 |
| Random seed | 1337 |
Optimizer configuration
Muon was used for the attention projections and shared-expert matrices. It used momentum 0.95, five Newton-Schulz iterations and weight decay 0.1.
AdamW handled the token and n-gram embeddings, router parameters, normalization parameters, scalar/gating parameters and routed experts.
AdamW used betas (0.9, 0.95), epsilon 1e-8 and weight decay 0.1. Embeddings used weight decay 0.01, while norms, gates and scalar parameters had no weight decay.
Muon was deliberately not used on the routed experts because it negatively affected routing balance and training speed in experiments.
Evaluation
The following benchmark results were obtained for the 100B-token checkpoint.
Scores are shown as percentages. acc is standard accuracy, while acc_norm is normalized accuracy.
| Benchmark | acc | acc_norm |
|---|---|---|
| HellaSwag | 32.82 | 38.36 |
| ARC-Easy | 51.18 | 48.11 |
| ARC-Challenge | 22.95 | 26.11 |
| ARC | 37.07 | 37.11 |
| PIQA | 67.41 | 66.92 |
| ArithMark3 | 37.30 | 37.40 |
Intelligence Index (acc_norm): 21.52
Quick start
The repository includes a native Transformers implementation with custom model code. You need trust_remote_code=True to load it.
Install the dependencies:
pip install -U transformers torch accelerate safetensors
Load the FP16 model:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "AuroraAI-Research/Aurora2-400M-A20M"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
repo,
subfolder="fp16",
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="auto",
).eval()
# <|bos|> (id 0) is prepended automatically
inputs = tokenizer("The Capital of France is", return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.5,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The repository contains three precision variants:
| Directory | Weights |
|---|---|
| Root | FP32 |
bf16/ |
BF16 |
fp16/ |
FP16 |
The tokenizer is stored in the repository root, so it does not need a subfolder argument.
Hardware note: The inference implementation uses BF16 activations on CUDA, even when the stored weights are FP16. A CUDA GPU with BF16 support is therefore recommended.
Inference notes
- Use batch size 1 or unpadded batches. The inference implementation does not reproduce the training-time per-document attention mask.
- Context length is 2,048 tokens.
- Beam search is not supported.
- Generation stops when the model produces BOS or EOS.
- The model is intended for causal text generation, not instruction-following.
Intended use and limitations
Aurora2 is primarily intended for research into small language models, sparse Mixture-of-Experts architectures, routing strategies, optimization and training efficiency. It can also serve as a starting point for further fine-tuning.
A few limitations are worth keeping in mind:
- Base model: It has not undergone instruction tuning or preference optimization. It is not designed to behave like a conversational assistant.
- Limited active capacity: Only about 19M parameters are active per token. Its factual knowledge, reasoning and long-range coherence are limited compared with substantially larger models.
- English-focused: Both the tokenizer and training mixture are primarily English-oriented.
- Short context: The model was trained with a 2,048-token context length.
- Training data: Web-scale datasets can contain inaccuracies, biases and offensive material. Outputs may reflect these issues.
- Downstream use: The model should not be used for decisions with real-world consequences. Check the licenses of individual training datasets before downstream use.
Citation
@misc{aurora2_400m_a20m,
title = {Aurora2-400M-A20M: A Small Sparse Mixture-of-Experts Language Model},
author = {AuroraAI-Research},
year = {2026},
note = {Pretrained on 100B tokens on a single GPU}
}
- Downloads last month
- 866