Kimi-K3.1-Pro EXL3 K2 - multi-core (MC)

moonshotai/Kimi-K3.1-Pro with its routed experts quantized to 2-bit EXL3, plus three choices of dense core in one repository: the original BF16 core and compact EXL3 cores at 8-bit and 6-bit. You select the core at load time; the routed experts are shared by all three views and stored once.

Built for cuteafd. Current serving status and measured layouts are listed below.

What "multi-core" means

A mixture-of-experts model has two very different kinds of weights:

  • Routed experts: most of the bytes (637.0 GiB, about 85.6% of this checkpoint's original view, including EXL3 scales and multipliers), but each token touches only a few of them. These expert weights are shared across views.
  • The dense core: attention, dense and shared feed-forward projections where present, and the output head. Every token runs through this core, so its encoding matters even though it occupies fewer bytes.

A multi-core (MC) checkpoint ships the experts once and the core several ways:

Core view Core format Total size Use it when
original (default) source-native BF16 core 743.9 GiB you want the source core without an additional core quantization
k8-compact EXL3 MCG 8-bit core 693.9 GiB you are testing a K8-capable runtime; this view is still Planned in cuteafd
k6-compact EXL3 MCG 6-bit core 681.4 GiB you want the smallest stored core

Sizes are tensor payloads from each view index's metadata.total_size, not runtime GPU requirements or the disk space of all alternate cores combined. Native tensors outside the selected core remain shared.

Each view has its own index, config and quantization metadata (model.safetensors.index.json / config.json / quantize_config.json for original, and the .k8. / .k6. siblings). cuteafd-mc.json lists the views. In cuteafd, CORE_VIEW=original|k8-compact|k6-compact selects one; the default is original. See the MC format.

How it was made

  • Source: moonshotai/Kimi-K3.1-Pro@6d0a0202956796543364d666ab19c6ad1a494ee1.
  • Experts: 247,296 routed projections at EXL3 MCG K2. The official source MXFP4 expert weights are decoded to BF16 for calibration before EXL3 encoding.
  • Compact cores: 926 selected attention, shared-expert, latent-MoE and vocabulary-head projections at K8 or K6.
  • Kept native: embeddings, the dense layer-0 FFN, routers and norms, recurrent KDA components, kv_b, attention-residual mixers, the DSA indexer, vision tower and multimodal projector. The native vision tensors are shared across views. No MTP or drafter weights are indexed in this package.
  • Calibration data: 1,426 prompts (1,201,086 Kimi tokens), using the checkpoint's tokenizer and chat template with thinking enabled. The front matter names wrldsuksgo2mars/agentic-quant-panel-v1, a token-budgeted calibration panel of real source code, worked mathematics, function-call schemas and multilingual text for MoE models used for coding, reasoning and tool use; the dataset revision and split are not recorded in this repository's listed metadata.
  • Method: the K2/K6 run quantizes attention and shared/latent entry projections, then experts, then shared/latent output projections, replaying serialized EXL3 weights reconstructed to BF16. The K8 core-only run instead propagates native source experts and reuses the stored K2 encodes. The DSA reference uses FP32 indexer math; its provenance explicitly does not emulate other engines' FP8 index-query/key quantization.
  • Quantizer: tpurtell/GPTQModel (main), 486cfc476ad42046d5382a11ffb8f856d82f4664 (K2 experts/K6 core); 1f920091232cc670ac67c233391d1a771f29f996 (K8 core).
  • Upstream status: Kimi source and K6 causal-core support are upstream (#3606, #3607, #3608). Upstream adds an explicit trust_remote_code=True opt-in for the pinned decoder; this is a loader-safety difference, not a new quantization recipe. The K8 core also needs the K1-K8 adapter widening on our fork (tpurtell/GPTQModel#5); upstream Kimi core validation still accepts only K6. That widening is not yet upstream.

Recipe

Setting Value
Codebook EXL3 MCG, multiplier 0xCBAC1FED
Bits experts K2; selected core K6 or K8
Seed / Hessian regularization (sigma) 787 / 0.025
Output scales automatic
Hessian signed block-Hadamard congruence; bounded 128-row FP32 products, FP64 outer sum, FP32 storage
Ordering K6: attention/shared-entry/latent-down, experts, shared-down/latent-up; K8: serialized core replay with native experts
Replay serialized EXL3, FP16 scales, reconstructed BF16
Rarely routed experts K2/K6 run: 1,024-row floor, ranks 17 and 32, rank-major then corpus-row; missing rows contribute identity
Calibration 1,426 prompts, 1,201,086 tokens; split not recorded

Provenance and validation receipts are in quantization-provenance.json, quantization-provenance.k8.json, package-provenance.json and package-validation.json; per-module formats and bit widths are in quantize_config*.json:tensor_storage.

Running it

Use a cuteafd build with MC core-view support; see the Kimi-K3.1-Pro family page and the README results tables.

On current cuteafd dev, the original BF16 and K6 compact views are qualified. K8 remains Planned, and whole-core FP8 is not supported for this MC original. The current benchmark matrix has an original-view card on 2 x RTX PRO 6000 + 6 x DGX Spark. It does not publish a one-RTX card for this checkpoint; fit depends on the selected view, KV pool and measured Spark capacity.

Speculation defaults off. External Inferact block dSpark is opt-in, with full-model served lossless qualification still pending (#2261); K3 DFlash2 compatibility is separately unqualified.

Other runtimes need EXL3 MCG support at the expert bit widths listed above and support for the selected core view and the family's preserved native components. The original view avoids additional target-core quantization; it does not remove the routed experts' EXL3 requirement.

License

Kimi-K3.1-Pro's own license applies; see LICENSE and the pinned license link in the front matter. This repository changes the weight encoding.

Downloads last month
53
Safetensors
Model size
420B params
Tensor type
BF16
·
F16
·
I16
·
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for wrldsuksgo2mars/Kimi-K3.1-Pro-EXL3-K2-MC

Quantized
(22)
this model

Dataset used to train wrldsuksgo2mars/Kimi-K3.1-Pro-EXL3-K2-MC