You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Access to Kimi-K3.1-Pro-DSpark is granted after manual review by the repository owner.

Log in or Sign Up to review the conditions and access this model content.

Kimi-K3.1-Pro-DSpark (ATOM / ROCm)

An MLA-native DSpark draft model for Kimi-K3.1 Pro, served through ATOM's dspark speculative method on 8 x MI355X: 5 dense layers with non-causal attention drafting 7 tokens per pass, a low-rank sequential Markov head, and a confidence head. The embedding and LM head are shared with the target.

  • 2,387,907,841 parameters, 4.78 GB on disk, serving window 32768.
  • Trained from random initialization for three epochs (56,367 steps) in BF16, on sequences of up to 32,768 tokens.

Performance

tok/fwd = 1 + accepted draft tokens / target forward steps: 1.0 at zero acceptance, at most 8 with 7 draft tokens. Same scale as the acceptance length on the reference card.

Reference draft: Inferact/Kimi-K3.1-Pro-DSpark-preview (revision 5e654fb), run by us on the same stack.

benchmark prompts this draft (ATOM) reference draft (ATOM) vs reference
GSM8K 1319 5.5191 5.4104 102.0%
MATH-500 500 4.2274 3.5658 118.6%
AIME 2026 30 3.2538 2.8494 114.2%
HumanEval 164 4.5646 4.0531 112.6%
MBPP 256 4.5504 4.2623 106.8%
MT-Bench 80 3.5085 3.3691 104.1%
SWE-bench Pro 128 3.8101 3.6671 103.9%
SPEED-Bench coding 80 5.2231 4.7908 109.0%
SPEED-Bench multilingual 80 3.9701 3.4456 115.2%
SPEED-Bench rag 80 3.9846 3.8240 104.2%
SPEED-Bench qa 80 3.6183 3.6627 98.8%
SPEED-Bench writing 80 3.1737 3.1196 101.7%
SPEED-Bench low-entropy (16k) 512 4.8879 4.6369 105.4%
mean (13 sets) 4.1763 3.8967 107.2%

The 13-set mean is 4.1763, 107.2% of the reference draft under the same protocol, engine and prompt files. That is an acceptance rate of 45.38% at 7 draft tokens, against 41.38% for the reference draft. This draft is ahead on 12 of 13 sets; SPEED-Bench qa is the exception (98.8%).

/!\ Both columns come from ATOM + 8 x MI355X with the same image and protocol. The reference draft needs three changes to run correctly on ATOM, and all three were applied for its column:

  • num_nextn_predict_layers: 5 added to its config.json; without it ATOM fails an assertion while sizing the draft KV pool.
  • rope_interleave set to false: ATOM's kimi_k3_dspark maps true to half-split rotation, while the checkpoint uses interleaved pairs.
  • Causal block attention, which its config.json declares (full_attention_causal), through a two-line patch to ATOM's dspark_proposer.py; stock ATOM drafts non-causally.

The reference card's own numbers use vLLM, sampling at temperature 1.0 / top_p 0.95, per-set max_tokens of 1,024-4,096 and the full MBPP and SWE-bench Pro sets, so they are not comparable with this table.

Raw results with accepted-length histograms are in benchmarks/ (this-draft--*, reference-draft--*).

Protocol

engine ATOM, TP=8, DCP=1, 8 x MI355X
software ATOM c63be4e4 (branch kimi-k31), AITER 607cb5e6f (ROCm/aiter dbaffe23ed + #7067), base image rocm/atom-dev@sha256:5332ae78c51985d1bd8462935bdd80817f12350c39f8e8db7d93ab9037ff5f1c
decoding greedy, concurrency 1, 7 speculative tokens, prefix caching off, max_num_seqs=8
window / batched tokens / max_tokens 16384 / 10240 / 12288; low-entropy 16k: 32768 / 32768 / 8192
prompts every prompt in each file, no sampling; SWE-bench Pro and low-entropy were split by line across servers and the counters summed
counters /debug/mtp_stats, differenced before/after each set

The 16384 window never binds: the longest prompt plus completion is 14,776 tokens (25,437 in the 32768-window set).


Quick Start

ATOM's Kimi-K3.1 support (c63be4e4 above) serves this BF16 checkpoint without draft-specific patches. The image behind the table also carries two ATOM patches; they affect only quantized drafts and vLLM plugin mode.

docker run -d --name atom-k31-dspark \
  --device=/dev/kfd --device=/dev/dri --group-add video \
  --security-opt seccomp=unconfined --cap-add=SYS_PTRACE \
  --ipc=host --shm-size 128g --network host \
  -e AITER_LOG_LEVEL=WARNING -e ATOM_MONO_ENABLE=0 -e AITER_CUSTOM_AR_MAX_SIZE=134217728 \
  -v /path/to/Kimi-K3.1:/target:ro -v /path/to/this/repo:/draft:ro \
  <atom-image-with-kimi-k3.1-support> \
  python -m atom.entrypoints.openai_server \
    --model /target --served-model-name Kimi-K3.1 \
    --method dspark --draft-model /draft --num-speculative-tokens 7 \
    --level 0 --cudagraph-mode FULL \
    --kv_cache_dtype bf16 --index-cache-dtype fp8 -tp 8 --trust-remote-code \
    --max-model-len 32768 --max-num-seqs 8 --max-num-batched-tokens 32768 \
    --gpu-memory-utilization 0.85 --block-size 128 \
    --no-enable_prefix_caching --server-port 8000
  • Keep num_nextn_predict_layers: 5 in config.json. ATOM sizes the draft KV pool from it and fails an assertion at startup without it.
  • /!\ Keep DCP at 1 with speculative decoding. Every number above uses DCP=1.

Weights

All tensors are BF16, in one shard (67 tensors).

part parameters
MLP (gate_proj, up_proj, down_proj) in all 5 layers 1.54B (64.6%)
MLA attention (q_a_proj, q_b_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj) in all 5 layers 0.51B (21.2%)
context_proj (5 target hidden states to the draft width) 0.26B (10.8%)
markov_head (rank 256) 0.08B (3.5%)
norms, confidence_head 0.1M

There is no embed_tokens or lm_head: the draft uses the target's.


Architecture

{
  "architectures": ["K3DSparkModel"],
  "model_type": "k3_dspark",
  "hidden_size": 7168,
  "intermediate_size": 14336,
  "num_hidden_layers": 5,
  "num_nextn_predict_layers": 5,
  "num_attention_heads": 64,
  "q_lora_rank": 1536,
  "kv_lora_rank": 512,
  "vocab_size": 163840,
  "target_layer_ids": [2, 23, 47, 71, 89],
  "mask_token_id": 163837,
  "markov_rank": 256,
  "enable_confidence_head": true,
  "rope_parameters": {
    "rope_type": "yarn", "factor": 32.0,
    "original_max_position_embeddings": 32768, "rope_theta": 50000.0
  }
}

Against the reference draft: 64 vs 96 attention heads, Markov rank 256 vs 512, target layers [2, 23, 47, 71, 89] vs [75, 79, 83, 87, 91], rope_theta 5e4 vs 1e6, and non-causal vs causal block attention.

/!\ Do not add rope_interleave to config.json. ATOM maps it to half-split rotation, while this draft uses interleaved pairs.

Downloads last month
-
Safetensors
Model size
2B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support