Kimi-K3.1-Pro EXL3 K2 - multi-core (MC)
moonshotai/Kimi-K3.1-Pro with its routed experts quantized to 2-bit EXL3, plus three choices of dense core in one repository: the original BF16 core and compact EXL3 cores at 8-bit and 6-bit. You select the core at load time; the routed experts are shared by all three views and stored once.
Built for cuteafd. Current serving status and measured layouts are listed below.
What "multi-core" means
A mixture-of-experts model has two very different kinds of weights:
- Routed experts: most of the bytes (637.0 GiB, about 85.6% of this checkpoint's
originalview, including EXL3 scales and multipliers), but each token touches only a few of them. These expert weights are shared across views. - The dense core: attention, dense and shared feed-forward projections where present, and the output head. Every token runs through this core, so its encoding matters even though it occupies fewer bytes.
A multi-core (MC) checkpoint ships the experts once and the core several ways:
| Core view | Core format | Total size | Use it when |
|---|---|---|---|
original (default) |
source-native BF16 core | 743.9 GiB | you want the source core without an additional core quantization |
k8-compact |
EXL3 MCG 8-bit core | 693.9 GiB | you are testing a K8-capable runtime; this view is still Planned in cuteafd |
k6-compact |
EXL3 MCG 6-bit core | 681.4 GiB | you want the smallest stored core |
Sizes are tensor payloads from each view index's metadata.total_size, not runtime GPU requirements or the disk space of all alternate cores combined. Native tensors outside the selected core remain shared.
Each view has its own index, config and quantization metadata (model.safetensors.index.json / config.json / quantize_config.json for original, and the .k8. / .k6. siblings). cuteafd-mc.json lists the views. In cuteafd, CORE_VIEW=original|k8-compact|k6-compact selects one; the default is original. See the MC format.
How it was made
- Source:
moonshotai/Kimi-K3.1-Pro@6d0a0202956796543364d666ab19c6ad1a494ee1. - Experts: 247,296 routed projections at EXL3 MCG K2. The official source MXFP4 expert weights are decoded to BF16 for calibration before EXL3 encoding.
- Compact cores: 926 selected attention, shared-expert, latent-MoE and vocabulary-head projections at K8 or K6.
- Kept native: embeddings, the dense layer-0 FFN, routers and norms, recurrent KDA components, kv_b, attention-residual mixers, the DSA indexer, vision tower and multimodal projector. The native vision tensors are shared across views. No MTP or drafter weights are indexed in this package.
- Calibration data: 1,426 prompts (1,201,086 Kimi tokens), using the checkpoint's tokenizer and chat template with thinking enabled. The front matter names wrldsuksgo2mars/agentic-quant-panel-v1, a token-budgeted calibration panel of real source code, worked mathematics, function-call schemas and multilingual text for MoE models used for coding, reasoning and tool use; the dataset revision and split are not recorded in this repository's listed metadata.
- Method: the K2/K6 run quantizes attention and shared/latent entry projections, then experts, then shared/latent output projections, replaying serialized EXL3 weights reconstructed to BF16. The K8 core-only run instead propagates native source experts and reuses the stored K2 encodes. The DSA reference uses FP32 indexer math; its provenance explicitly does not emulate other engines' FP8 index-query/key quantization.
- Quantizer: tpurtell/GPTQModel (
main),486cfc476ad42046d5382a11ffb8f856d82f4664(K2 experts/K6 core);1f920091232cc670ac67c233391d1a771f29f996(K8 core). - Upstream status: Kimi source and K6 causal-core support are upstream (#3606, #3607, #3608). Upstream adds an explicit
trust_remote_code=Trueopt-in for the pinned decoder; this is a loader-safety difference, not a new quantization recipe. The K8 core also needs the K1-K8 adapter widening on our fork (tpurtell/GPTQModel#5); upstream Kimi core validation still accepts only K6. That widening is not yet upstream.
Recipe
| Setting | Value |
|---|---|
| Codebook | EXL3 MCG, multiplier 0xCBAC1FED |
| Bits | experts K2; selected core K6 or K8 |
| Seed / Hessian regularization (sigma) | 787 / 0.025 |
| Output scales | automatic |
| Hessian | signed block-Hadamard congruence; bounded 128-row FP32 products, FP64 outer sum, FP32 storage |
| Ordering | K6: attention/shared-entry/latent-down, experts, shared-down/latent-up; K8: serialized core replay with native experts |
| Replay | serialized EXL3, FP16 scales, reconstructed BF16 |
| Rarely routed experts | K2/K6 run: 1,024-row floor, ranks 17 and 32, rank-major then corpus-row; missing rows contribute identity |
| Calibration | 1,426 prompts, 1,201,086 tokens; split not recorded |
Provenance and validation receipts are in quantization-provenance.json, quantization-provenance.k8.json, package-provenance.json and package-validation.json; per-module formats and bit widths are in quantize_config*.json:tensor_storage.
Running it
Use a cuteafd build with MC core-view support; see the Kimi-K3.1-Pro family page and the README results tables.
On current cuteafd dev, the original BF16 and K6 compact views are qualified. K8 remains Planned, and whole-core FP8 is not supported for this MC original. The current benchmark matrix has an original-view card on 2 x RTX PRO 6000 + 6 x DGX Spark. It does not publish a one-RTX card for this checkpoint; fit depends on the selected view, KV pool and measured Spark capacity.
Speculation defaults off. External Inferact block dSpark is opt-in, with full-model served lossless qualification still pending (#2261); K3 DFlash2 compatibility is separately unqualified.
Other runtimes need EXL3 MCG support at the expert bit widths listed above and support for the selected core view and the family's preserved native components. The original view avoids additional target-core quantization; it does not remove the routed experts' EXL3 requirement.
License
Kimi-K3.1-Pro's own license applies; see LICENSE and the pinned license link in the front matter. This repository changes the weight encoding.
- Downloads last month
- 53
Model tree for wrldsuksgo2mars/Kimi-K3.1-Pro-EXL3-K2-MC
Base model
moonshotai/Kimi-K3.1-Pro