DCore-DSpark-Qwen3-8B-Budget8

Draft model for speculative decoding with Qwen/Qwen3-8B as the target model, trained with a draft budget of 8 tokens per block.

DSpark baseline: DFlash + Markov head + confidence head, trained by us for comparison in the DCore paper.

This is a draft model: it is conditioned on hidden states of the target model and is not meant to be used as a standalone language model.

Paper: arXiv:2610.24993

Model Details

Item Value
Target model Qwen/Qwen3-8B (36 layers)
Draft architecture DSparkDraftModel
Draft layers 5
Hidden size / heads / KV heads 4096 / 32 / 8
Block size 7
Target layers used as context features 1, 9, 17, 25, 33
markov_head_type vanilla
markov_rank 256
enable_confidence_head True
confidence_head_with_markov True
Vocabulary 151936 (Qwen3 tokenizer)
Precision bfloat16

With budget 8, DSpark drafts 7 tokens per block (block size 7).

Files

  • README.md
  • config.json
  • dflash.py
  • dspark.py
  • markov_head.py
  • model.safetensors

Usage

from transformers import AutoModel

draft = AutoModel.from_pretrained(
    "ernie-research/DCore-DSpark-Qwen3-8B-Budget8",
    trust_remote_code=True,
    dtype="bfloat16",
)

The draft model must be paired with Qwen/Qwen3-8B as the target model in a speculative decoding framework that supports DFlash-style block drafting.

Collection

Part of the DCore collection by ERNIE Research.

Downloads last month
154
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ernie-research/DCore-DSpark-Qwen3-8B-Budget8

Finetuned
Qwen/Qwen3-8B
Finetuned
(2,364)
this model

Collection including ernie-research/DCore-DSpark-Qwen3-8B-Budget8

Paper for ernie-research/DCore-DSpark-Qwen3-8B-Budget8