DCore-Domino-Qwen3-4B-Budget16

Draft model for speculative decoding with Qwen/Qwen3-4B as the target model, trained with a draft budget of 16 tokens per block.

Domino baseline: DFlash architecture with a Domino projector, trained by us for comparison in the DCore paper.

This is a draft model: it is conditioned on hidden states of the target model and is not meant to be used as a standalone language model.

Paper: arXiv:2610.24993

Model Details

Item Value
Target model Qwen/Qwen3-4B (36 layers)
Draft architecture DFlashDraftModel
Draft layers 5
Hidden size / heads / KV heads 2560 / 32 / 8
Block size 16
Target layers used as context features 1, 9, 17, 25, 33
projector_type domino
emb_dim 256
gru_hidden_dim 1024
Vocabulary 151936 (Qwen3 tokenizer)
Precision bfloat16

Files

  • config.json
  • dflash.py
  • domino.py
  • model.safetensors
  • registry.py

Usage

from transformers import AutoModel

draft = AutoModel.from_pretrained(
    "wingchiuloong/DCore-Domino-Qwen3-4B-Budget16",
    trust_remote_code=True,
    dtype="bfloat16",
)

The draft model must be paired with Qwen/Qwen3-4B as the target model in a speculative decoding framework that supports DFlash-style block drafting.

Collection

Part of the DCore collection by ERNIE Research.

Downloads last month
68
Safetensors
Model size
0.6B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for wingchiuloong/DCore-Domino-Qwen3-4B-Budget16

Finetuned
Qwen/Qwen3-4B
Finetuned
(1,304)
this model

Collection including wingchiuloong/DCore-Domino-Qwen3-4B-Budget16

Paper for wingchiuloong/DCore-Domino-Qwen3-4B-Budget16