Aurora2-400M-A20M

Aurora2-400M-A20M is a decoder-only language model with approximately 400M total parameters and 19M active parameters per token. It uses a top-1 Mixture-of-Experts (MoE) architecture with 32 routed experts and one shared expert per layer.

The model was pretrained from scratch on 100B tokens using a single GPU. It is a base model, not an instruction-tuned assistant: there was no supervised instruction-tuning or RLHF stage.

Property Value
Total parameters 399,597,454
Active parameters per token 19,050,382
Experts 32 routed + 1 shared per layer
Layers 12
Hidden size 192
Context length 2,048 tokens
Vocabulary 16,384
Training tokens 100B
Training precision BF16 compute, FP32 master weights
Hardware Single RTX Pro 6000, approximately 36 hours

Architecture

Aurora2 uses a relatively narrow attention stack paired with large, sparse feed-forward layers. Each token is routed to one of 32 experts, alongside the always-active shared expert.

Core configuration

Component Configuration
Hidden size 192
Layers 12
Attention heads 6 query / 2 KV
Attention type Grouped-query attention (3:1)
Head dimension 32
Positional encoding RoPE, base 10,000
Normalization Pre-RMSNorm, FP32 QK normalization
Attention FlexAttention, causal and per-document masking
Routed experts 32 per layer, SwiGLU, FFN dimension 1,776
Shared expert 1 per layer, SwiGLU, FFN dimension 256
Routing FP32 softmax, top-1, dropless
Output head Tied token embeddings
Logit soft-capping 30

The router scales each selected expert's output by its routing probability. There is no expert capacity factor or token dropping.

Router training includes a Switch-style load-balancing loss (coefficient 0.01) and router z-loss (coefficient 0.001).

Additional components

  • Hashed n-gram embeddings: Causal hashes of the last 2, 3 and 4 token IDs index three separate 16,384-bucket embedding tables. Their outputs are concatenated, projected to the hidden dimension and added to the token embeddings through a learned per-channel gate.
  • Embedding skip connections: Every block combines its input with the original embedding using learned coefficients, initialized to [1, 0].
  • Value residuals: Layers after the first mix their value tensors with the first layer's values, using learned coefficients initialized to [0.5, 0.5].
  • Per-head attention gates: Each attention head has a learned output gate derived from the first 12 channels of the normalized block input. It is initialized to produce a multiplier of 1.
  • Learned router bias: Each expert has a trainable bias added to its routing logits.

Tokenizer

Aurora2 uses a byte-level BPE tokenizer trained from scratch.

  • Vocabulary size: 16,384.
  • Every digit is an individual token ("123" becomes three tokens).
  • Special tokens:
    • <|bos|>: ID 0
    • <|eos|>: ID 1
    • <|pad|>: ID 2
  • The tokenizer automatically prepends <|bos|> to encoded text.

During training, documents were packed together, with each document starting with a BOS token. No EOS was inserted between documents. BOS tokens also served as document boundaries for attention masking.

Training data

Training data was streamed from the Hugging Face Hub, tokenized on the fly and packed into sequences of 2,048 tokens. The tokenizer was trained on approximately 1B characters from the same data mixture.

Only training splits were used. ClimbMix shard 6542 was reserved for validation; benchmark evaluation splits were not used for training.

Dataset Purpose Average token share Approx. epochs
ClimbMix General web text 85.8% < 1
FineMath-4+ Mathematical web text 6.0% < 1
Wikipedia (English) Encyclopedic text 3.0% < 1
SmolTalk Chat and QA 2.5% ~2.5
OpenMathInstruct-2 Mathematical QA 1.5% ~2
NuminaMath-CoT Mathematical QA 1.2% ~3

Staged data mixture

The mixture was adjusted throughout training. Math and QA/chat sources received greater weight during the final decay phase, while ClimbMix remained the majority source.

Training stage ClimbMix FineMath Wikipedia SmolTalk OpenMath Numina
0–60B tokens 89.3% 5.4% 2.7% 1.25% 0.75% 0.6%
60–80B tokens 85.8% 6.0% 3.0% 2.5% 1.5% 1.2%
80–100B tokens 75.3% 7.8% 3.9% 6.25% 3.75% 3.0%

Instruction-style datasets were converted into plain-text formats, including Question: ... Answer: ..., Problem: ... Solution: ... and User: ... Assistant: .... These examples were part of pretraining; they do not constitute an instruction-tuning stage.

Training procedure

Hyperparameter Value
Total training tokens 100B
Sequences per step 128
Sequence length 2,048
Tokens per step 262,144
Approximate optimizer steps 381,000
Learning-rate schedule Warmup-Stable-Decay (WSD)
Warmup 1B tokens, linear
Stable phase 1B–80B tokens
Decay Cosine decay over the final 20B tokens
Peak learning rate 1.5e-3
Router learning-rate multiplier 0.5
Optimizer Muon + AdamW
Gradient clipping 1.0
Random seed 1337

Optimizer configuration

Muon was used for the attention projections and shared-expert matrices. It used momentum 0.95, five Newton-Schulz iterations and weight decay 0.1.

AdamW handled the token and n-gram embeddings, router parameters, normalization parameters, scalar/gating parameters and routed experts.

AdamW used betas (0.9, 0.95), epsilon 1e-8 and weight decay 0.1. Embeddings used weight decay 0.01, while norms, gates and scalar parameters had no weight decay.

Muon was deliberately not used on the routed experts because it negatively affected routing balance and training speed in experiments.

Evaluation

The following benchmark results were obtained for the 100B-token checkpoint.

Scores are shown as percentages. acc is standard accuracy, while acc_norm is normalized accuracy.

Benchmark acc acc_norm
HellaSwag 32.82 38.36
ARC-Easy 51.18 48.11
ARC-Challenge 22.95 26.11
ARC 37.07 37.11
PIQA 67.41 66.92
ArithMark3 37.30 37.40

Intelligence Index (acc_norm): 21.52

Quick start

The repository includes a native Transformers implementation with custom model code. You need trust_remote_code=True to load it.

Install the dependencies:

pip install -U transformers torch accelerate safetensors

Load the FP16 model:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AuroraAI-Research/Aurora2-400M-A20M"

tokenizer = AutoTokenizer.from_pretrained(repo)

model = AutoModelForCausalLM.from_pretrained(
    repo,
    subfolder="fp16",
    trust_remote_code=True,
    torch_dtype=torch.float16,
    device_map="auto",
).eval()

# <|bos|> (id 0) is prepended automatically
inputs = tokenizer("The Capital of France is", return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.5,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The repository contains three precision variants:

Directory Weights
Root FP32
bf16/ BF16
fp16/ FP16

The tokenizer is stored in the repository root, so it does not need a subfolder argument.

Hardware note: The inference implementation uses BF16 activations on CUDA, even when the stored weights are FP16. A CUDA GPU with BF16 support is therefore recommended.

Inference notes

  • Use batch size 1 or unpadded batches. The inference implementation does not reproduce the training-time per-document attention mask.
  • Context length is 2,048 tokens.
  • Beam search is not supported.
  • Generation stops when the model produces BOS or EOS.
  • The model is intended for causal text generation, not instruction-following.

Intended use and limitations

Aurora2 is primarily intended for research into small language models, sparse Mixture-of-Experts architectures, routing strategies, optimization and training efficiency. It can also serve as a starting point for further fine-tuning.

A few limitations are worth keeping in mind:

  • Base model: It has not undergone instruction tuning or preference optimization. It is not designed to behave like a conversational assistant.
  • Limited active capacity: Only about 19M parameters are active per token. Its factual knowledge, reasoning and long-range coherence are limited compared with substantially larger models.
  • English-focused: Both the tokenizer and training mixture are primarily English-oriented.
  • Short context: The model was trained with a 2,048-token context length.
  • Training data: Web-scale datasets can contain inaccuracies, biases and offensive material. Outputs may reflect these issues.
  • Downstream use: The model should not be used for decisions with real-world consequences. Check the licenses of individual training datasets before downstream use.

Citation

@misc{aurora2_400m_a20m,
  title  = {Aurora2-400M-A20M: A Small Sparse Mixture-of-Experts Language Model},
  author = {AuroraAI-Research},
  year   = {2026},
  note   = {Pretrained on 100B tokens on a single GPU}
}
Downloads last month
866
Safetensors
Model size
0.4B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AuroraAI-Research/Aurora2-400M-A20M

Finetunes
1 model

Datasets used to train AuroraAI-Research/Aurora2-400M-A20M