North.ML 1 Smol
Collection
The full series of north.ml 1 series. Finished. Uses custom architecture and may be hard to convert. Use at your own risk. • 3 items • Updated • 1
Shared SentencePiece tokenizer for the North Star model family:
Vocabulary size: 32,000
Format: SentencePiece (.model file)
import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file="tokenizer.model")
ids = sp.encode("Hello world", out_type=int)