Qwen3.5-397B-A17B — MXFP4 experts + FP8 attention + BF16 MTP

File-to-file quantization of Qwen/Qwen3.5-397B-A17B with AMD Quark (RTN, no calibration). The MTP (NEXTN draft head) is kept in BF16 so the speculative draft is exact, which maximizes accept length.

Quantization scheme

Component Precision
Routed + shared experts MXFP4 (W4A4, OCP MX, group size 32)
Attention (q/k/v/o proj) FP8 E4M3 (per-channel weight + dynamic per-token activation)
GDN / linear-attn, lm_head, MoE gates BF16
MTP / NEXTN draft head BF16

Size: ~236 GB (≈3.2× smaller than the 752 GB BF16 master).

A companion uniform-FP8-MTP variant is at thehosy/Qwen3.5-397B-A17B-MXFP4-MTP-FP8 (−7 GB, but requires an SGLang patch to load the quantized MTP correctly).

Serving (SGLang on ROCm / gfx950, e.g. 2× MI350X)

SGLANG_USE_AITER=1 SGLANG_USE_AITER_UNIFIED_ATTN=1 \
python -m sglang.launch_server \
  --model-path thehosy/Qwen3.5-397B-A17B-MXFP4-MTP-BF16 \
  --trust-remote-code --tp-size 2 \
  --attention-backend aiter --linear-attn-backend triton \
  --speculative-algorithm EAGLE --speculative-num-steps 3 \
  --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
  --page-size 64 --mem-fraction-static 0.85

No code changes needed for this BF16-MTP variant (the draft loads natively in BF16).

Benchmarks (2× MI350X, TP2, aiter backend)

  • Accuracy (512 questions, thinking-off): GSM8K 0.967, MMLU 0.867 (spec decoding is lossless — same outputs as no-spec).
  • Speculative decoding (EAGLE / NEXTN, BF16 MTP): GSM8K mean accept ≈ 3.3 (per-position agreement p1/p2/p3 ≈ 0.94/0.84/0.76).
  • Throughput (GSM8K, spec on): 230 / 524 / 1418 tok/s at concurrency 1 / 4 / 16 (TPOT 3.8 / 4.9 / 7.0 ms; TTFT 96 / 527 / 593 ms).

Numbers are hardware/workload specific; accept length is strongly workload-dependent.

License

Inherits the license of the base model Qwen/Qwen3.5-397B-A17B.

Downloads last month
27
Safetensors
Model size
222B params
Tensor type
U8
·
F32
·
BF16
·
F8_E4M3
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for thehosy/Qwen3.5-397B-A17B-MXFP4-MTP-BF16

Quantized
(81)
this model