DSpark v2 head, full vocab: 2.62x vs no drafter, 4.67 tok/step

README.md

@@ -0,0 +1,170 @@
+---
+license: apache-2.0
+base_model:
+  - Qwen/Qwen3.6-27B
+library_name: transformers
+tags:
+  - speculative-decoding
+  - dspark
+  - draft-model
+  - vllm
+  - qwen3
+datasets:
+  - nvidia/Open-SWE-Traces
+  - greghavens/gpt-5.6-sol-coding-and-debugging-traces
+  - greghavens/fable-5-coding-and-debugging-traces
+  - greghavens/glm-5.2-coding-and-debugging-traces
+  - Roman1111111/claude-sonnet-4.6-120000x
+  - qsardor/Claude-Sonnet-Opus
+  - ianncity/GLM-5.2-Conversation
+---
+
+# Qwen3.6-27B-DSpark
+
+A DSpark draft head for **Qwen3.6-27B**, trained on a mixed agentic + general-chat
+corpus. Drops into stock vLLM as a `--speculative-config`.
+
+**2.62× faster decoding than no speculation**, **1.48× faster than the upstream
+DSpark head**, and **1.27× faster than Qwen3.6's built-in MTP**, at **4.67 tokens
+accepted per target pass**. Measured over 96 held-out prompts spanning agentic
+tool-use, coding and general chat, with generation length and sampling fixed so
+wall-clock is directly comparable.
+
+> **Two builds of this head.** This is the **full-vocabulary** build (8.20 GiB) —
+> prefer it for long-context agentic work. A **vocabulary-truncated** build of the
+> same trained weights is at
+> [`abstract-extraordinary/Qwen3.6-27B-DSpark-49k`](https://huggingface.co/abstract-extraordinary/Qwen3.6-27B-DSpark-49k)
+> (6.20 GiB, faster on short prompts, no throughput advantage on long ones). See
+> [Vocabulary variants](#vocabulary-variants).
+
+## Usage
+
+```bash
+vllm serve unsloth/Qwen3.6-27B-NVFP4 \
+  --speculative-config '{"method":"dspark",
+                         "model":"abstract-extraordinary/Qwen3.6-27B-DSpark",
+                         "num_speculative_tokens":8}' \
+  --trust-remote-code
+```
+
+Use `num_speculative_tokens: 8`. The head drafts blocks of exactly 8 tokens
+(`block_size=8`), so a smaller value leaves trained capacity unused and a larger one
+asks it to predict positions it has never seen.
+
+## Benchmarks
+
+**96 held-out prompts across 8 workload buckets.** All five configurations ran in a
+single session against the same prompts, identical but for `--speculative-config`,
+each emitting exactly 300 greedy tokens (`temperature=0`, `min_tokens=max_tokens`,
+`ignore_eos`) so wall-clock is directly comparable. Target `unsloth/Qwen3.6-27B-NVFP4`,
+single GB10, concurrency 1. Every figure is a draft-weighted pool over the whole run,
+not a mean over prompts.
+
+| config | tok/s | tok/step | accept | vs no drafter |
+|---|---|---|---|---|
+| no drafter | 10.05 | — | — | 1.00× |
+| [`satgeze/Qwen3.6-27B-DSpark`](https://huggingface.co/satgeze/Qwen3.6-27B-DSpark) (upstream, untrained) | 17.74 | 2.86 | 23.4% | 1.77× |
+| Qwen3.6 built-in MTP (3 tokens) | 20.76 | 3.25 | 75.1% | 2.07× |
+| **this model** | **26.28** | **4.67** | 46.2% | **2.62×** |
+| [same head, 49k vocab](https://huggingface.co/abstract-extraordinary/Qwen3.6-27B-DSpark-49k) | 28.18 | 4.66 | 46.0% | 2.80× |
+
+The upstream head was run at **both** `num_speculative_tokens: 8` and `15` (its own
+`block_size`); 8 was faster (17.74 vs 17.07 tok/s) and is quoted here, so the
+baseline is shown at its stronger setting.
+
+MTP shows a much higher acceptance *percentage* while being slower. The two are not
+in tension: MTP drafts 3 tokens per pass and this head drafts 8, so a higher hit
+rate on fewer guesses still yields fewer accepted tokens per step (3.25 vs 4.67).
+Tokens per step is the quantity that converts into wall-clock.
+
+By workload:
+
+| bucket | mean prompt tok | no drafter | this model | speedup |
+|---|---|---|---|---|
+| general chat (ianncity) | 125 | 11.10 | 34.51 | 3.11× |
+| agentic (kimi-k3) | 1,000 | 10.92 | 33.11 | 3.03× |
+| agentic (fable-5) | 1,446 | 10.84 | 32.62 | 3.01× |
+| general chat (qsardor) | 68 | 11.11 | 32.13 | 2.89× |
+| general/code/math (sonnet-4.6) | 93 | 11.11 | 30.95 | 2.79× |
+| agentic (glm-5.2) | 5,770 | 9.78 | 23.48 | 2.40× |
+| agentic (gpt-5.6-sol) | 9,752 | 8.92 | 20.59 | 2.31× |
+| agentic SWE (nvidia) | 15,841 | 7.82 | 16.67 | 2.13× |
+
+Absolute tok/s tracks prompt length rather than head quality — long prompts spend
+most of their wall-clock in prefill, which speculation does not accelerate. The
+speedup column is the like-for-like comparison. The kimi-k3 bucket was excluded from
+training entirely and is a clean generalisation probe.
+
+### Acceptance by draft position
+
+![Cumulative acceptance by draft position](acceptance-by-position.png)
+
+Acceptance decays *multiplicatively*: surviving to position 3 means winning four
+consecutive rejection tests. That compounding is why a few points at position 0 turn
+into a large difference in tokens per step, and it is what separates this head from
+the untrained upstream one. Upstream starts at a respectable 62.4% at position 0 but
+halves roughly every two positions to 5.8% by position 7; this head starts at 82.7%
+and still retains 22.8%.
+
+## Vocabulary variants
+
+| | this model | [`-49k`](https://huggingface.co/abstract-extraordinary/Qwen3.6-27B-DSpark-49k) |
+|---|---|---|
+| draft vocabulary | 248,320 (full) | 49,167 |
+| size on disk | 8.20 GiB | 6.20 GiB |
+| tok/s (96 prompts) | 26.28 | 28.18 |
+| tok/step | 4.67 | 4.66 |
+| best for | long-context agentic | short-prompt / chat-shaped |
+
+Both are the **same trained weights**. The `-49k` build row-gathers `lm_head` and
+`markov_w2` to the tokens the target actually emits — a post-hoc transform, not a
+retrain. It is faster per decode step because the draft head streams its whole
+weight set every step and a smaller head streams less. That saving is large on short
+prompts and **disappears on long ones**, where prefill dominates wall-clock, while a
+small acceptance cost remains. **If your prompts run to thousands of tokens, use
+this full-vocab build.**
+
+## Training
+
+Assistant turns **regenerated by the target model itself** (temperature 0.7, top-p
+0.95), so the head learns the distribution it will actually have to predict — not the
+distribution of whatever model produced the original traces.
+
+14,707 conversations from seven sources: fresh `nvidia/Open-SWE-Traces` windows, three
+agentic trace sets, and three general-chat sets. 44.93M supervised tokens at 46.6% loss
+density (mean 3,055 per sample). 600 steps in 6.3 h on one node, lr 6e-4, 384 anchors,
+`block_size=8`; final loss 0.73 from ~1.05. Held-out offline accepted length was
+monotone across all 20 checkpoints and plateaued at steps 540–600, so the run
+converged rather than being truncated.
+
+Held-out prompts were excluded from training by content fingerprint before harvesting,
+train/val shard offsets verified disjoint, and the publishers' own splits respected
+where they exist. All evaluation above uses only held-out samples.
+
+## Limitations
+
+- **Correctness is unverified — not failed.** Speculative decoding is
+  distribution-preserving under rejection sampling, but we could not run a
+  token-identity check: greedy decoding is not reproducible on this stack run-to-run
+  (a target-only run matches *itself* on 3 of 12 prompts, because prefix caching and
+  batch composition change the numerical path). This head adds no measurable
+  divergence beyond that baseline, which is weaker evidence than a proof.
+- **No live-traffic measurement.** Every number here is a held-out benchmark at
+  concurrency 1. Held-out and live figures have disagreed before for this line of
+  heads.
+- **Single hardware.** Benchmarked only on one GB10 with `unsloth/Qwen3.6-27B-NVFP4`.
+  Speculative-decoding economics are bandwidth-dependent; expect different ratios
+  elsewhere.
+- Acceptance is far from uniform across workloads (40.5–60.4% by bucket). Long agentic
+  prompts draft best per step but gain least in wall-clock, because prefill dominates
+  there.
+- Run-to-run spread on this benchmark is about 1% on pooled tok/s (measured by
+  repeating the no-drafter and MTP configurations on two vLLM builds). Differences
+  smaller than that should not be read as real.
+
+## Credits
+
+Initialised from [`satgeze/Qwen3.6-27B-DSpark`](https://huggingface.co/satgeze/Qwen3.6-27B-DSpark).
+DSpark architecture from *DSpark: Confidence-Scheduled Speculative Decoding with
+Semi-Autoregressive Generation*. Training data from NVIDIA and the community dataset
+authors listed above.

config.json

@@ -0,0 +1,68 @@
+{
+  "architectures": [
+    "Qwen3DSparkModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_output_gate": true,
+  "block_size": 8,
+  "bos_token_id": 248044,
+  "confidence_head_with_markov": true,
+  "dtype": "bfloat16",
+  "enable_confidence_head": true,
+  "eos_token_id": 248044,
+  "full_attention_interval": 4,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 17408,
+  "layer_types": [
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention",
+    "full_attention"
+  ],
+  "linear_conv_kernel_dim": 4,
+  "linear_key_head_dim": 128,
+  "linear_num_key_heads": 16,
+  "linear_num_value_heads": 48,
+  "linear_value_head_dim": 128,
+  "mamba_ssm_dtype": "float32",
+  "markov_head_type": "vanilla",
+  "markov_rank": 256,
+  "mask_token_id": 248200,
+  "max_position_embeddings": 262144,
+  "max_window_layers": 28,
+  "model_type": "qwen3",
+  "mtp_num_hidden_layers": 1,
+  "mtp_use_dedicated_embeddings": false,
+  "num_anchors": 384,
+  "num_attention_heads": 32,
+  "num_hidden_layers": 5,
+  "num_key_value_heads": 8,
+  "num_target_layers": 64,
+  "output_gate_type": "swish",
+  "pad_token_id": null,
+  "partial_rotary_factor": 1.0,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "partial_rotary_factor": 1.0,
+    "rope_theta": 10000000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "target_layer_ids": [
+    1,
+    16,
+    31,
+    46,
+    61
+  ],
+  "tie_word_embeddings": false,
+  "transformers_version": "5.14.1",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 248320
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f0432e860cdd51dbbc4ab848af74e66383b08240d2f27bf4706f52e3405c215
+size 8800317146