Ling-3.0-Tiny-DSpark

Version: v1

Experimental release. An independent community DSpark draft for Ling-3.0-tiny, not an official inclusionAI release or a standalone LLM. Trained with SpecForge; evaluated using SGLang. Research use, not production certification.

Created by Jay Yun — GitHub · LinkedIn.

I built an open-source DSpark training platform, with its public source release coming soon. The project supports offline training on 1× H100 and online training on 2× H100. These project training configurations are distinct from this checkpoint's latest 3,000-step continuation and reported evaluation, which ran on 1× H200. The platform repository link will be added when published.

Checkpoint

Property	Value
Target	inclusionAI/Ling-3.0-tiny
Evaluated target revision	a2ee06c0f2de5b171701aee7f73f70a1da75483b
Architecture	Qwen3DSparkModel; 3 full-attention layers
Hidden / intermediate size	1536 / 4608
Attention	16 query / 16 KV heads; head dimension128
Feature layers	3, 7, 11, 15, 19
Mask ID	156901
Vanilla Markov rank	512
Draft proposals / verification width	7 / 8 including bonus
Weight dtype	BF16
Stored draft parameters	274,217,729 (~274M)
Evaluated context limit	4096

Model SHA-256: e9e4ad541c6d73f3d02488e18afc5f9b7d5089c81b953a630c808f927700d4a6. Configured maximum positions131072 does not establish validated long-context support.

Acceptance length and throughput

One H200;20 exact-chat prompts; max_tokens256; concurrency1; temperature0; reasoning off; seed42; static verify-all. Target repeatability was checked.

Metric	Prior blend	v1 candidate
acc_len including bonus	2.775398	2.786241
Acceptance rate	25.3628%	25.5177%
Correct / proposed drafts	2901 /11438	2908 /11396
Verification rounds	1634	1628
Output tokens/sec	63.3469	63.8391
Accepted draft tokens/sec	40.6480	41.0626
Exact-text parity vs target	20/20	20/20

Target-only output:26.9769tok/s. Observed candidate speedup~2.37x is a single-run pilot result, not a repeated or production-wide claim. The0.391% acc_len gain missed our3% improvement screen. Goal acc_len5–6 was not reached.

Acceptance rate=correct/proposed; acc_len=1+correct/verification rounds. The1 represents the bonus token. Cached prefix and training tau are different metrics.

Position	Prefix survival	Conditional acceptance
1	66.15%	66.15%
2	41.15%	62.21%
3	26.78%	65.07%
4	18.30%	68.35%
5	12.04%	65.77%
6	8.42%	69.90%
7	5.77%	68.61%

Exact-count histogram for0–7 accepted proposals: [551,407,234,138,102,59,43,94]. These bins are not position acceptance probabilities.

Training and lineage

Original Tiny50K draft →75% Tiny /25% Flash Markov output-bias blend →3,000-step full-draft continuation. Rank512 concatenates factors representing the output mixture; it does not average factor coordinates or transplant the Flash backbone. Flash-target acceptance cannot be inherited as Tiny-target acceptance.

Continuation data:2,048 existing Tiny-t0 responses plus512 additional prompts regenerated by Tiny at temperature0, reasoning off, tagged regenerated_by=ling-t0. All512 succeeded;2,560 cache samples, zero skips. Source/domain proportions follow the previous pilot. Held-out exclusion uses normalized exact prompt hashes and IDs, not semantic or pretraining decontamination. Raw data is not a release asset.

Full-draft3000steps, LR2e-5, seed44, gamma4; CE0.1+L1(2TV)0.9+confidence1, acceptance auxiliary coefficient0. Training/cache length3072; evaluation4096. Because data and horizon changed together, improvement cannot be attributed solely to the additional prompts.

Serving and validation scope

Recorded image: lmsysorg/sglang:dev-cu12-Ling-3.0-tiny (mutable tag). FA3; deterministic inference; CUDA graphs/radix cache disabled; context4096; SGLANG_RAGGED_VERIFY_MODE=static; DSPARK block7 and8 draft-token slots. Use the Tiny target tokenizer/chat template with thinking explicitly disabled. Do not load this draft alone as a Transformers text-generation model.

The following recipe matches the recorded serving flags. The mutable image tag has not been freshly re-tested for this release. Current stock SGLang, other GPUs, quantization, vLLM, llama.cpp and thinking mode have not been validated for this candidate. No GPU was rented for publication.

Not yet tested: independent fresh64 long1024-token gate, concurrency4/8, repeated timings. No standardized GSM8K/MATH/HumanEval result is claimed here.20/20 parity is evidence for this pilot only, not a universal correctness guarantee.

hf download inclusionAI/Ling-3.0-tiny \
  --revision a2ee06c0f2de5b171701aee7f73f70a1da75483b --local-dir ./ling-tiny
hf download jayyun98/Ling-3.0-Tiny-DSpark --local-dir ./ling-tiny-dspark

docker run --rm --gpus all --ipc=host \
  -p 127.0.0.1:30000:30000 -v "$PWD:/models:ro" \
  -e SGLANG_RAGGED_VERIFY_MODE=static \
  lmsysorg/sglang:dev-cu12-Ling-3.0-tiny \
  python3 -m sglang.launch_server \
  --model-path /models/ling-tiny --trust-remote-code \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path /models/ling-tiny-dspark \
  --speculative-dspark-block-size 7 --speculative-num-draft-tokens 8 \
  --served-model-name ling-tiny --reasoning-parser ling3 \
  --attention-backend fa3 --context-length 4096 \
  --max-running-requests 1 --max-total-tokens 8192 \
  --chunked-prefill-size -1 --disable-radix-cache --disable-cuda-graph \
  --enable-deterministic-inference --host 0.0.0.0 --port 30000

curl http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"ling-tiny","messages":[{"role":"user","content":"Explain speculative decoding briefly."}],"temperature":0,"top_p":1,"seed":42,"max_tokens":256,"chat_template_kwargs":{"enable_thinking":false}}'

License and attribution

Tiny's upstream card declares MIT. The Flash DSpark donor card declares other without explicit redistribution terms in the inspected card. This repository retains license: other; it makes no additional MIT or commercial-use grant for the donor-derived weights. Consult the upstream terms for your intended use.

Upstream sources: Tiny target and Flash DSpark donor.

Credit: inclusionAI, the DSpark authors, SpecForge and SGLang contributors. Card organization references RadixArk/Qwen3.8-27B-DSpark. Its benchmark results are not ours and are not directly comparable.