lfm3-804m-xfer
A ~976M-parameter LFM3 student approximating LiquidAI's ~1.7B
LFM3-1.7B teacher through
weight-space transfer and search โ activation-PCA basis projection,
lifted weights, checkpoint soups, and zeroth-order coefficient search.
The root model.safetensors is the current champion
(soup4u_cmaes, see results below). Raw pipeline checkpoints live in
checkpoints/; the full experiment log and code are in xfer-forge/.
Architecture
Same tri-hybrid layer pattern as the teacher (conv / sliding-attention /
full-attention, 24 layers), hidden 1344 (vs teacher 2048), untied
embeddings (tie_word_embeddings=false โ input embedding and decode
head are separate matrices, +172M params), custom selected-pair
sliding-attention RoPE frequencies (stored as inv_freq_override in
config.json โ requires the bundled modeling_lfm3.py /
configuration_lfm3.py, i.e. trust_remote_code=True).
Results (mean CE, nats/token โ lower is better)
Sacred held-out banks: rep_fw = fineweb @25M offset, rep_dom =
domain_v2 @60M, rep_code = code @40M. Fit banks (fit50/fit50b) were
found to anti-correlate with rep_fw and are reported for reference only.
| checkpoint | rep_fw | rep_dom | rep_code | notes |
|---|---|---|---|---|
| soup4u_cmaes (root weights) | 3.7972 | 2.3300 | 2.1502 | champion: untied grouped soup, best 3-bank mean + best policy-KL |
| soup4_cmaes | 3.7981 | 2.3299 | 2.1513 | tied grouped soup r3 |
| soup3_cmaes | 3.7990 | 2.3310 | 2.1518 | first block-structured soup |
| soup2_cmaes | 3.7897 | 2.3388 | 2.1665 | previous champion; best fineweb CE |
| franken_soup_cmaes | 3.7880 | 2.3423 | 2.1719 | best fineweb; greedy per-layer merge |
| layermix1 | 3.7891 | 2.3530 | 2.1778 | fw-specialist |
| decode_tied_soup2 | โ | โ | โ | untied head member (ridge-refit decode head) |
Policy-KL (teacherโstudent, rep_fw/64): soup4u KL 1.3696 / top1 0.4891 โ best of all checkpoints (soup2: 1.3737/0.4883). On rep_dom/64: soup4 KL 0.9994 ~ soup4u 0.9997 (soup2: 1.0065). Teacher CE headroom: rep_fw 3.018 vs student ~3.72.
The weight-space search has essentially saturated: all viable
checkpoints cluster within ~0.02 CE and the whole zoo is a single
error-correlated basin (per-window CE corr โฅ0.81). See
xfer-forge/WEIGHT_SPACE_PROGRAM.md for the full experiment log โ
including the second research round (NEXT_FRONTIER*.md): Fisher
merging, closed-form decode-head refits (all variants), RoPE audits,
CKA layer alignment, and the untied-head soup that produced this
champion.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
"Akahsizrr/lfm3-804m-xfer", trust_remote_code=True,
torch_dtype="bfloat16").cuda().eval()
tok = AutoTokenizer.from_pretrained("Akahsizrr/lfm3-804m-xfer")
The untied head means lm_head.weight is stored separately โ no action
needed, trust_remote_code=True handles it (tie_word_embeddings=false
in config.json).
- Downloads last month
- 33