lfm3-804m-xfer

A ~976M-parameter LFM3 student approximating LiquidAI's ~1.7B LFM3-1.7B teacher through weight-space transfer and search โ€” activation-PCA basis projection, lifted weights, checkpoint soups, and zeroth-order coefficient search. The root model.safetensors is the current champion (soup4u_cmaes, see results below). Raw pipeline checkpoints live in checkpoints/; the full experiment log and code are in xfer-forge/.

Architecture

Same tri-hybrid layer pattern as the teacher (conv / sliding-attention / full-attention, 24 layers), hidden 1344 (vs teacher 2048), untied embeddings (tie_word_embeddings=false โ€” input embedding and decode head are separate matrices, +172M params), custom selected-pair sliding-attention RoPE frequencies (stored as inv_freq_override in config.json โ€” requires the bundled modeling_lfm3.py / configuration_lfm3.py, i.e. trust_remote_code=True).

Results (mean CE, nats/token โ€” lower is better)

Sacred held-out banks: rep_fw = fineweb @25M offset, rep_dom = domain_v2 @60M, rep_code = code @40M. Fit banks (fit50/fit50b) were found to anti-correlate with rep_fw and are reported for reference only.

checkpoint rep_fw rep_dom rep_code notes
soup4u_cmaes (root weights) 3.7972 2.3300 2.1502 champion: untied grouped soup, best 3-bank mean + best policy-KL
soup4_cmaes 3.7981 2.3299 2.1513 tied grouped soup r3
soup3_cmaes 3.7990 2.3310 2.1518 first block-structured soup
soup2_cmaes 3.7897 2.3388 2.1665 previous champion; best fineweb CE
franken_soup_cmaes 3.7880 2.3423 2.1719 best fineweb; greedy per-layer merge
layermix1 3.7891 2.3530 2.1778 fw-specialist
decode_tied_soup2 โ€” โ€” โ€” untied head member (ridge-refit decode head)

Policy-KL (teacherโ†’student, rep_fw/64): soup4u KL 1.3696 / top1 0.4891 โ€” best of all checkpoints (soup2: 1.3737/0.4883). On rep_dom/64: soup4 KL 0.9994 ~ soup4u 0.9997 (soup2: 1.0065). Teacher CE headroom: rep_fw 3.018 vs student ~3.72.

The weight-space search has essentially saturated: all viable checkpoints cluster within ~0.02 CE and the whole zoo is a single error-correlated basin (per-window CE corr โ‰ฅ0.81). See xfer-forge/WEIGHT_SPACE_PROGRAM.md for the full experiment log โ€” including the second research round (NEXT_FRONTIER*.md): Fisher merging, closed-form decode-head refits (all variants), RoPE audits, CKA layer alignment, and the untied-head soup that produced this champion.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained(
    "Akahsizrr/lfm3-804m-xfer", trust_remote_code=True,
    torch_dtype="bfloat16").cuda().eval()
tok = AutoTokenizer.from_pretrained("Akahsizrr/lfm3-804m-xfer")

The untied head means lm_head.weight is stored separately โ€” no action needed, trust_remote_code=True handles it (tie_word_embeddings=false in config.json).

Downloads last month
33
Safetensors
Model size
1.0B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support