EvoLenTokenizer/base-200k-sft-checkpoints

Baseline BPE-200K — SFT checkpoints (all 56 tasks)

Fine-tuned classification checkpoints from the baseline BPE pretrained model at step 200,000 (vocab 5,120), for use as the control against EvoLenTokenizer/evolen-200k-sft-checkpoints.

Base model: EvoLenTokenizer/base-200k. Tokenizer: standard BPE, vocab 5,120 — not the EvoLen tokenizer.

Complete and hyperparameter-matched

All 56 tasks are published. Every checkpoint uses the same learning rate, weight decay, warmup ratio, epochs, seed, model_max_length, precision and checkpoint-selection rule as its EvoLen-200k counterpart, so the only differences are the tokenizer and the pretrained model.

37 of these came from the original baseline runs. The remaining 19 — all of mouse/*, all of tf/*, seven EMP/*, splice/reconstructed and allchr_csv — were re-run specifically to match EvoLen's configuration; several of those had no baseline run at all, and allchr_csv had previously been run at model_max_length=200 where EvoLen used 512.

Nine GBM tasks were originally trained at per_device_batch_size=64 against EvoLen's 128; coverage.csv records the config used for every task.

A second asymmetry to be aware of

Even for the 28 matched tasks, the selection procedure differs. EvoLen-200k reports the maximum test MCC over a sweep of 18–119 configurations per task; the baseline is a single run at one configuration. Comparing a max-over-many against a single draw favours EvoLen independently of tokenizer quality. Treat the deltas accordingly.

Layout

One folder per task, <SUITE>/<task>/, each with model.safetensors, config.json, tokenizer.json, tokenizer_config.json, special_tokens_map.json, trainer_state.json and training_args.bin. Optimizer and scheduler state are stripped.

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
    "EvoLenTokenizer/base-200k-sft-checkpoints",
    subfolder="NT/H3K27ac", trust_remote_code=True)

coverage.csv lists all 56 tasks with their status (published / batch-size mismatch / hyperparameter mismatch / no run) and both configs side by side.

Reproducing

Evaluation only. fp16 autocast, eval batch size from training_args.bin, per-task model_max_length from training_args.bin, and sorted(set(...)) label ordering for multiclass tasks. Tokenize with padding="longest", truncation=True; attention mask is input_ids.ne(pad_token_id); metric is sklearn.metrics.matthews_corrcoef over argmax predictions.

Environment: Python 3.9.18, transformers 4.35.2, scikit-learn 1.6.1, torch 2.8.0, numpy 2.0.2, tokenizers 0.15.2, accelerate 0.25.0.