nanochat-keith-sft

This is a custom-trained nanochat model uploaded from checkpoint: sft/d20 at step 715.

Model Details

• Architecture: GPT-style transformer
• Training Stage: SFT
• Model Tag: d20
• Training Step: 715

Model Configuration

{
  "sequence_len": 2048,
  "vocab_size": 65536,
  "n_layer": 20,
  "n_head": 10,
  "n_kv_head": 10,
  "n_embd": 1280
}

Training Configuration

{}

Performance Metrics

• Validation BPB: N/A
• Validation Loss: 1.1147265434265137

Usage

import torch
from nanochat.checkpoint_manager import build_model
from nanochat.tokenizer import get_tokenizer
from nanochat.engine import Engine

# Download and load the model
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load model (you'll need to download the checkpoint files)
checkpoint_dir = "./checkpoint"  # directory with model files
step = 715
model, tokenizer, meta = build_model(checkpoint_dir, step, device, phase="eval")

# Create engine for generation
engine = Engine(model, tokenizer)

# Generate text
prompt = "Hello, how are you?"
tokens = tokenizer(prompt, prepend="<|bos|>")
samples, _ = engine.generate_batch(tokens, num_samples=1, max_tokens=100, temperature=0.8)
print(tokenizer.decode(samples[0]))

Training Details

This model was trained using the nanochat framework by Andrej Karpathy.

Training Data

• Custom SFT data: keith_identity.jsonl, keith_reputation.jsonl
• Base datasets: SmolTalk, MMLU, GSM8K, ARC, SpellingBee
• Training stage: sft

Computational Requirements

Training was performed following the nanochat speedrun pipeline.

Citation

If you use this model, please cite the original nanochat project:

@misc{nanochat,
  author = {Karpathy, Andrej},
  title = {nanochat: A minimalistic ChatGPT clone},
  year = {2024},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

License

This model follows the MIT license of the nanochat project.

Checkpoint Lineage and Retention

This repository is the SFT checkpoint in the private lineage:

1. keithtyser/nanochat-keith — base checkpoint
2. keithtyser/nanochat-keith-mid — intermediate checkpoint
3. keithtyser/nanochat-keith-sft — supervised-fine-tuned checkpoint

Keep all three while reproducibility, regression comparison, rollback, or training resume matters. Stage-specific data claims in the older card text should be treated as provisional unless they are backed by a dataset manifest and training log; the datasets themselves are not bundled here.