DCore-Qwen3-4B-Budget8

Draft model of DCore for speculative decoding with Qwen/Qwen3-4B as the target model, trained with a draft budget of 8 tokens per block.

DCore = DFlare (draft architecture) + D-Pace (training recipe).

This is a draft model: it is conditioned on hidden states of the target model and is not meant to be used as a standalone language model.

Paper: arXiv:2610.24993

Model Details

Item Value
Target model Qwen/Qwen3-4B (36 layers)
Draft architecture DFlareDraftModel
Draft layers 5
Hidden size / heads / KV heads 2560 / 32 / 8
Block size (budget) 8
Target layers used as context features 1, 5, 9, 13, 17, 21, 25, 29, 33
Mask token id 151669
Vocabulary 151936 (Qwen3 tokenizer)
Precision bfloat16
Training data PerfectBlend

DFlare is a DFlash variant with two structural changes:

  1. Cross-attention uses separate k/v projections for the context (target hidden states) and for the draft tokens.
  2. Multi-layer target hidden states are fused with a learnable layer_fusion_weights[D, T] matrix instead of a single Linear(T*H -> H).

Files

  • config.json
  • dflare.py
  • dflash.py
  • model.safetensors
  • registry.py

Usage

from transformers import AutoModel

draft = AutoModel.from_pretrained(
    "wingchiuloong/DCore-Qwen3-4B-Budget8",
    trust_remote_code=True,
    dtype="bfloat16",
)

The draft model must be paired with Qwen/Qwen3-4B as the target model in a speculative decoding framework that supports DFlash-style block drafting.

Collection

Part of the DCore collection by ERNIE Research.

Downloads last month
115
Safetensors
Model size
0.5B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for wingchiuloong/DCore-Qwen3-4B-Budget8

Finetuned
Qwen/Qwen3-4B
Finetuned
(1,304)
this model

Collection including wingchiuloong/DCore-Qwen3-4B-Budget8

Paper for wingchiuloong/DCore-Qwen3-4B-Budget8