Qwen3.5-397B-A17B — MXFP4 experts + FP8 attention + BF16 MTP
File-to-file quantization of Qwen/Qwen3.5-397B-A17B with AMD Quark (RTN, no calibration). The MTP (NEXTN draft head) is kept in BF16 so the speculative draft is exact, which maximizes accept length.
Quantization scheme
| Component | Precision |
|---|---|
| Routed + shared experts | MXFP4 (W4A4, OCP MX, group size 32) |
| Attention (q/k/v/o proj) | FP8 E4M3 (per-channel weight + dynamic per-token activation) |
GDN / linear-attn, lm_head, MoE gates |
BF16 |
| MTP / NEXTN draft head | BF16 |
Size: ~236 GB (≈3.2× smaller than the 752 GB BF16 master).
A companion uniform-FP8-MTP variant is at thehosy/Qwen3.5-397B-A17B-MXFP4-MTP-FP8
(−7 GB, but requires an SGLang patch to load the quantized MTP correctly).
Serving (SGLang on ROCm / gfx950, e.g. 2× MI350X)
SGLANG_USE_AITER=1 SGLANG_USE_AITER_UNIFIED_ATTN=1 \
python -m sglang.launch_server \
--model-path thehosy/Qwen3.5-397B-A17B-MXFP4-MTP-BF16 \
--trust-remote-code --tp-size 2 \
--attention-backend aiter --linear-attn-backend triton \
--speculative-algorithm EAGLE --speculative-num-steps 3 \
--speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
--page-size 64 --mem-fraction-static 0.85
No code changes needed for this BF16-MTP variant (the draft loads natively in BF16).
Benchmarks (2× MI350X, TP2, aiter backend)
- Accuracy (512 questions, thinking-off): GSM8K 0.967, MMLU 0.867 (spec decoding is lossless — same outputs as no-spec).
- Speculative decoding (EAGLE / NEXTN, BF16 MTP): GSM8K mean accept ≈ 3.3 (per-position agreement p1/p2/p3 ≈ 0.94/0.84/0.76).
- Throughput (GSM8K, spec on): 230 / 524 / 1418 tok/s at concurrency 1 / 4 / 16 (TPOT 3.8 / 4.9 / 7.0 ms; TTFT 96 / 527 / 593 ms).
Numbers are hardware/workload specific; accept length is strongly workload-dependent.
License
Inherits the license of the base model Qwen/Qwen3.5-397B-A17B.
- Downloads last month
- 27
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for thehosy/Qwen3.5-397B-A17B-MXFP4-MTP-BF16
Base model
Qwen/Qwen3.5-397B-A17B