Instructions to use ZhuoBohang-AMD/Kimi-K3.1-Pro-DSpark with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ZhuoBohang-AMD/Kimi-K3.1-Pro-DSpark with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ZhuoBohang-AMD/Kimi-K3.1-Pro-DSpark", device_map="auto") - Notebooks
- Google Colab
- Kaggle
You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Access to Kimi-K3.1-Pro-DSpark is granted after manual review by the repository owner.
Log in or Sign Up to review the conditions and access this model content.
Kimi-K3.1-Pro-DSpark (ATOM / ROCm)
An MLA-native DSpark draft model for Kimi-K3.1 Pro, served through ATOM's dspark
speculative method on 8 x MI355X: 5 dense layers with non-causal attention drafting 7 tokens per pass, a low-rank
sequential Markov head, and a confidence head. The embedding and LM head are shared with the target.
- 2,387,907,841 parameters, 4.78 GB on disk, serving window 32768.
- Trained from random initialization for three epochs (56,367 steps) in BF16, on sequences of up to 32,768 tokens.
Performance
tok/fwd = 1 + accepted draft tokens / target forward steps: 1.0 at zero acceptance, at most 8 with 7 draft
tokens. Same scale as the acceptance length on the reference card.
Reference draft: Inferact/Kimi-K3.1-Pro-DSpark-preview
(revision 5e654fb), run by us on the same stack.
| benchmark | prompts | this draft (ATOM) | reference draft (ATOM) | vs reference |
|---|---|---|---|---|
| GSM8K | 1319 | 5.5191 | 5.4104 | 102.0% |
| MATH-500 | 500 | 4.2274 | 3.5658 | 118.6% |
| AIME 2026 | 30 | 3.2538 | 2.8494 | 114.2% |
| HumanEval | 164 | 4.5646 | 4.0531 | 112.6% |
| MBPP | 256 | 4.5504 | 4.2623 | 106.8% |
| MT-Bench | 80 | 3.5085 | 3.3691 | 104.1% |
| SWE-bench Pro | 128 | 3.8101 | 3.6671 | 103.9% |
| SPEED-Bench coding | 80 | 5.2231 | 4.7908 | 109.0% |
| SPEED-Bench multilingual | 80 | 3.9701 | 3.4456 | 115.2% |
| SPEED-Bench rag | 80 | 3.9846 | 3.8240 | 104.2% |
| SPEED-Bench qa | 80 | 3.6183 | 3.6627 | 98.8% |
| SPEED-Bench writing | 80 | 3.1737 | 3.1196 | 101.7% |
| SPEED-Bench low-entropy (16k) | 512 | 4.8879 | 4.6369 | 105.4% |
| mean (13 sets) | 4.1763 | 3.8967 | 107.2% |
The 13-set mean is 4.1763, 107.2% of the reference draft under the same protocol, engine and prompt files. That is an acceptance rate of 45.38% at 7 draft tokens, against 41.38% for the reference draft. This draft is ahead on 12 of 13 sets; SPEED-Bench qa is the exception (98.8%).
/!\ Both columns come from ATOM + 8 x MI355X with the same image and protocol. The reference draft needs three changes to run correctly on ATOM, and all three were applied for its column:
num_nextn_predict_layers: 5added to itsconfig.json; without it ATOM fails an assertion while sizing the draft KV pool.rope_interleaveset tofalse: ATOM'skimi_k3_dsparkmapstrueto half-split rotation, while the checkpoint uses interleaved pairs.- Causal block attention, which its
config.jsondeclares (full_attention_causal), through a two-line patch to ATOM'sdspark_proposer.py; stock ATOM drafts non-causally.
The reference card's own numbers use vLLM, sampling at temperature 1.0 / top_p 0.95, per-set max_tokens of 1,024-4,096 and the full MBPP and SWE-bench Pro sets, so they are not comparable with this table.
Raw results with accepted-length histograms are in benchmarks/ (this-draft--*, reference-draft--*).
Protocol
| engine | ATOM, TP=8, DCP=1, 8 x MI355X |
| software | ATOM c63be4e4 (branch kimi-k31), AITER 607cb5e6f (ROCm/aiter dbaffe23ed + #7067), base image rocm/atom-dev@sha256:5332ae78c51985d1bd8462935bdd80817f12350c39f8e8db7d93ab9037ff5f1c |
| decoding | greedy, concurrency 1, 7 speculative tokens, prefix caching off, max_num_seqs=8 |
| window / batched tokens / max_tokens | 16384 / 10240 / 12288; low-entropy 16k: 32768 / 32768 / 8192 |
| prompts | every prompt in each file, no sampling; SWE-bench Pro and low-entropy were split by line across servers and the counters summed |
| counters | /debug/mtp_stats, differenced before/after each set |
The 16384 window never binds: the longest prompt plus completion is 14,776 tokens (25,437 in the 32768-window set).
Quick Start
ATOM's Kimi-K3.1 support (c63be4e4 above) serves this BF16 checkpoint without draft-specific patches. The image
behind the table also carries two ATOM patches; they affect only quantized drafts and vLLM plugin mode.
docker run -d --name atom-k31-dspark \
--device=/dev/kfd --device=/dev/dri --group-add video \
--security-opt seccomp=unconfined --cap-add=SYS_PTRACE \
--ipc=host --shm-size 128g --network host \
-e AITER_LOG_LEVEL=WARNING -e ATOM_MONO_ENABLE=0 -e AITER_CUSTOM_AR_MAX_SIZE=134217728 \
-v /path/to/Kimi-K3.1:/target:ro -v /path/to/this/repo:/draft:ro \
<atom-image-with-kimi-k3.1-support> \
python -m atom.entrypoints.openai_server \
--model /target --served-model-name Kimi-K3.1 \
--method dspark --draft-model /draft --num-speculative-tokens 7 \
--level 0 --cudagraph-mode FULL \
--kv_cache_dtype bf16 --index-cache-dtype fp8 -tp 8 --trust-remote-code \
--max-model-len 32768 --max-num-seqs 8 --max-num-batched-tokens 32768 \
--gpu-memory-utilization 0.85 --block-size 128 \
--no-enable_prefix_caching --server-port 8000
- Keep
num_nextn_predict_layers: 5inconfig.json. ATOM sizes the draft KV pool from it and fails an assertion at startup without it. - /!\ Keep DCP at 1 with speculative decoding. Every number above uses DCP=1.
Weights
All tensors are BF16, in one shard (67 tensors).
| part | parameters |
|---|---|
MLP (gate_proj, up_proj, down_proj) in all 5 layers |
1.54B (64.6%) |
MLA attention (q_a_proj, q_b_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj) in all 5 layers |
0.51B (21.2%) |
context_proj (5 target hidden states to the draft width) |
0.26B (10.8%) |
markov_head (rank 256) |
0.08B (3.5%) |
norms, confidence_head |
0.1M |
There is no embed_tokens or lm_head: the draft uses the target's.
Architecture
{
"architectures": ["K3DSparkModel"],
"model_type": "k3_dspark",
"hidden_size": 7168,
"intermediate_size": 14336,
"num_hidden_layers": 5,
"num_nextn_predict_layers": 5,
"num_attention_heads": 64,
"q_lora_rank": 1536,
"kv_lora_rank": 512,
"vocab_size": 163840,
"target_layer_ids": [2, 23, 47, 71, 89],
"mask_token_id": 163837,
"markov_rank": 256,
"enable_confidence_head": true,
"rope_parameters": {
"rope_type": "yarn", "factor": 32.0,
"original_max_position_embeddings": 32768, "rope_theta": 50000.0
}
}
Against the reference draft: 64 vs 96 attention heads, Markov rank 256 vs 512, target layers
[2, 23, 47, 71, 89] vs [75, 79, 83, 87, 91], rope_theta 5e4 vs 1e6, and non-causal vs causal block attention.
/!\ Do not add rope_interleave to config.json. ATOM maps it to half-split rotation, while this draft uses
interleaved pairs.
- Downloads last month
- -