---
language:
- en
license: mit
library_name: pytorch
tags:
- text-generation
- causal-lm
- minimind
- pretrained
---

# MiniMind-135M

A 135M parameter decoder-only transformer (GPT-2 style) pretrained on OpenWebText from scratch.

## Model Details

| Attribute | Value |
|-----------|-------|
| Parameters | 134.6M |
| Vocab size | 32,000 |
| Hidden dim | 768 |
| Layers | 12 |
| Attention heads | 12 |
| Context window | 512 |
| Training data | OpenWebText (~615M tokens) |
| Training steps | 100,000 |
| Best val loss | 4.0174 |
| Tokenizer | Custom ByteLevel BPE (32k) |

## Usage

```python
from transformers import AutoTokenizer
import torch

# Load tokenizer
tok = AutoTokenizer.from_pretrained("achavan1211/minimind-135m")

# Load model (requires MiniMind class definition)
model = MiniMind(32000, 768, 12, 12, 512, 0.1)
state_dict = torch.load("best_model.pt", map_location="cpu")
model.load_state_dict(state_dict["model_state_dict"])
model.eval()

# Generate
prompt = "The future of artificial intelligence"
ids = tok.encode(prompt, return_tensors="pt")
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=100, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))
```

## Training

- **Optimizer**: AdamW (lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)
- **Schedule**: Warmup (2k steps) + cosine decay to 3e-5
- **Effective batch**: 48 (12 × 4 gradient accumulation)
- **Hardware**: A100 GPU
- **Duration**: ~2.5 days

## Limitations

This is a **base model** (no instruction fine-tuning). It generates text continuations,
not conversational responses. For assistant-like behavior, it needs further fine-tuning
on instruction data.