---
license: mit
base_model: XiaomiMiMo/MiMo-V2.6-Flash-MOPD
base_model_relation: quantized
library_name: gguf
pipeline_tag: text-generation
language:
- de
- en
tags:
- sakura
- sakura-micro
- gguf
- llama.cpp
- moe
- mimo
- expert-pruning
- ssd-streaming
- strix-halo
- experimental
---
# Sakura — MiMo-V2.6-Flash-MOPD P160 (expert-pruned, SSD-streaming ready)
Sakura MiMo P160 is an experimental expert-pruned GGUF of Xiaomi's MiMo-V2.6-Flash-MOPD: 160 of 256 routed experts
per MoE layer, about 53.79 GiB. It is built to run on a single 64 GB AMD Strix Halo (Radeon 8060S, Vulkan) with
~35 GiB on the GPU and the remaining experts computed from system RAM — the **MiMo SSD Streaming** setup described
below. Focus: German instructions, long planning prompts and coding. Independent community build, **not** an official
Xiaomi release.
Part of the **[Sakura Micro](https://huggingface.co/collections/webmp3/sakura-micro-6aba74331f2e996ba1268a92)** line:
compact, locally runnable derivatives of large models (lines:
[Sakura](https://huggingface.co/collections/webmp3/sakura-6aba73f6376c82b665cf2ea2),
[Sakura Mini](https://huggingface.co/collections/webmp3/sakura-mini-6aba73f6a5a41296a3540373),
[Sakura Micro](https://huggingface.co/collections/webmp3/sakura-micro-6aba74331f2e996ba1268a92)).
## Provenance
- Base model: [XiaomiMiMo/MiMo-V2.6-Flash-MOPD](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD), the MOPD2
upgrade of MiMo-V2.6-Flash-RL (309B total / 15B active, 48 layers, 256 experts, top-8). Its routers are
bit-identical to the RL checkpoint, so an expert selection measured on RL transfers exactly.
- Built from Xiaomi's original weights: only the kept experts were downloaded (HTTP ranges), the MXFP4 experts were
repacked losslessly, FP8 tensors converted exactly like llama.cpp's converter (checked byte-identical against the
reference on the RL release).
- Expert selection (per layer): protect the experts carrying 80 % of the full model's routing on its **own German
answers** to long coding/planning prompts (on-policy measurement), then the decode-trace hot experts, then REAP
saliency bands and routed-token share decide. Selections tuned only on foreign text scored better perplexity but
looped in chat; measuring on the model's own outputs fixed that.
- Quantization: llama.cpp with an importance matrix (Baekpica's, cut to the kept experts). Experts IQ2_XXS/IQ2_XS in
layers 12–35, IQ2_XS/IQ2_S elsewhere, down projections of layers 44/46/47 IQ3_XXS; attention IQ4_XS, dense FFN
Q6_K, embeddings Q8_0, output Q6_K.
## Release artifact
`MiMo-V2.6-Flash-MOPD-P160-OnPolicy-v3la-IQ2_XS.gguf`
- Quantization: mixed low-bit, mostly IQ2_XS experts (details above); `IQ2_XS` in the file name is the dominant type
- Size: 57,761,339,136 Bytes (53.79 GiB)
- SHA-256: `90dbd32b5d41ec902da81d0c369a8649cb29c059d7d9624a0c2f49511fab3b1b`
- Parameters: ~195B total after pruning (from 309B), ~15B active per token (unchanged)
- The bundled `SHA256SUMS` file can be used to verify the download.
## Measured results
| Build | Size | PPL German | PPL code | Chat checks |
|---|---:|---:|---:|---|
| **Sakura MiMo P160 (this release, MOPD)** | **53.8 GiB** | **6.02** | **5.80** | short 7/8 · German stress 6/6 · long texts 4/4 · roadmaps 2/2 at temperature 1.0 · no loops |
| same selection and recipe, RL checkpoint | 53.8 GiB | 6.02 | 5.87 | roadmap: 1/2 at temperature 0 (rumination loop), 2/2 at 0.6 |
| 144 experts, RL (cut from TrevorJS IQ2) | 53.5 GiB | 6.12 | 6.10 | short 7/8 · long 4/4 · roadmaps 2/2 at temperature 0 |
| full MiMo-V2.6-Flash-RL (TrevorJS IQ2, reference) | 91.9 GiB | 5.04 | 4.23 | — |
Decode speed of this release: **9–10.5 tok/s at 16k context** and 5.6–7.7 tok/s at 32k, measured with the desktop in
normal parallel use (details below).
Perplexity: held-out German and code text, 8 × 512 tokens, not used for selection or calibration (lower is better).
Speed: AMD Ryzen AI Max+ 395 / Radeon 8060S, 64 GB, Vulkan, MiMo SSD Streaming setup below, measured with the desktop
in normal parallel use (browser, chat apps; free RAM near zero) — see the speed table below for the context-size effect.
Chat checks: short factual/arithmetic prompts, long free-form texts, six German mid-length answers and two long German
roadmap/deployment prompts with fixed closing lines. These are small functional checks, not a claim of general
quality versus the original model.
## Why 160 experts
We built and tested several expert counts from the same MiMo-V2.6-Flash family on this machine
(held-out PPL German / code; lower is better):
| Experts kept per layer | Size | PPL German | PPL code | Chat |
|---|---:|---:|---:|---|
| 128 (hot-expert selection) | 48.0 GiB | 14.50 | 7.30 | German broken (word salad, loops) |
| 128 (German-protected) | 48.0 GiB | 5.97 | 6.99 | German fine, but code loops and format errors |
| 144 (on-policy) | 53.5 GiB | 6.12 | 6.10 | stable |
| **160 (on-policy, this release)** | **53.8 GiB** | **6.02** | **5.80** | stable, no loops at temperature 1.0 |
| 256 (full model, reference) | 91.9 GiB | 5.04 | 4.23 | — (2.6 tok/s here, mostly paged from NVMe) |
The 128/144-expert rows and the full-model reference are MiMo-V2.6-Flash-RL builds. The same 160-expert selection on
the RL checkpoint scored 6.02 / 5.87, so the step from 144 to 160 helps on its own; MOPD adds a little on code.
Smaller REAP/pruning levels cost noticeable quality — at 128 experts German or code breaks down. 160 experts give
the best quality among the variants that still run at a usable speed on a 64 GB machine: the file is barely larger
than the 144-expert build because the middle layers use slightly smaller quant types. Keeping more experts would
push more weights off the GPU and cost speed. 160 is the sweet spot **among the variants we tested** for 64 GB;
we did not build 176+.
## Running it with stock llama.cpp
Any recent llama.cpp with `mimo2` support loads this GGUF. With enough memory for the whole file on the GPU:
```
llama-server -m MiMo-V2.6-Flash-MOPD-P160-OnPolicy-v3la-IQ2_XS.gguf -ngl 99 --jinja -c 32768 --temp 1.0 --top-p 0.95
```
Use temperature 1.0 / top-p 0.95 (Xiaomi's recommendation): at 0–0.6, long thinking tasks sometimes fell into
rumination loops in our tests; at 1.0 they did not.
## MiMo SSD Streaming — running it with less VRAM
The model is larger than a 32 GB GPU. MiMo SSD Streaming keeps the attention, the outlier-heavy early layers and the
late layers on the GPU and lets the CPU compute the experts of a block of middle layers (14–33, ~20 GiB) from RAM,
with the NVMe as backing store. Components (all in `runtime/` of this repo):
| Component | What it does |
|---|---|
| `-ot "…ffn_(gate\|up\|down)_exps\.weight=CPU"` | keeps the experts of chosen layers in host memory (mmap); fits the model into ~35 GiB GPU |
| Swap-MoE `--expert-streaming` ([ek15072809/Swap-MoE](https://github.com/ek15072809/Swap-MoE)) + our GPU guard patch | demand-pages those experts from NVMe instead of loading them all up front |
| Op-offload **on** (no `--no-op-offload`) | prompt processing copies the used experts to the GPU (fast when the experts are in RAM) |
| Expert lock (`LLAMA_EXPERT_LOCK_LIST`, our patch, optional) | pins the most-used CPU-side experts in RAM (`VirtualLock`) so multi-token batches do not page-fault |
| UTF-8 sanitizing in chat parsing (our patch) | pruned models sometimes emit a broken emoji byte sequence; stock llama-server then fails with HTTP 500 — the reply is kept, broken bytes become U+FFFD |
**Speed on this machine (decode, tok/s; desktop in parallel use, free RAM near zero):**
| Mode | Context | Lock | Decode | Prefill (4k-token prompt) |
|---|---:|---|---:|---:|
| **fast (default)** | 16k | none | **9–10.5** | 13–25 |
| long | 32k | hottest 30 % (~6 GiB) | 5.6–7.7 | 14–18 |
| (24k, for reference) | 24k | none | 6.6–7.4 | 32 |
All rows use the default `-ub 512`, which gives the best overall speed. A larger ubatch trades decode for prefill:
| `-ub` | Prefill (2k prompt) | Decode |
|---:|---:|---:|
| **512 (default)** | 12.7 | **9.6** |
| 1024 | 22.3 | 5.9 |
| 2048 | 34–39 (4k prompt) | ~5.3 |
When the CPU-side experts do not fit in RAM they are read from the NVMe once per ubatch, so larger ubatches speed up
prefill; but their bigger GPU buffers spill into shared system memory and take RAM away from those experts, which
nearly halves decode. A larger ubatch only pays off when the prompt is more than about twice as long as the answer
(for example a one-off read of large files); for coding with long reasoning keep 512. `-ub 4096` ran out of GPU
memory on this machine.
The context size matters because the KV cache and GPU buffers spill into shared system memory; at 16k enough RAM
stays free for the CPU-side experts. On a quiet machine with the RAM to spare, the same runtime reached 8.9 tok/s
decode, ~120 tok/s prefill and 15/18/17 tok/s on 2/4/8-token batches with the expert lock (measured with our
144-expert build). n-gram speculation (`--spec-type ngram-mod`) helped the RL-based builds when returning edited files,
but not this MOPD build (about 50 % acceptance), so it is off by default.
Example for 64 GB Strix Halo (`runtime/start_server_example.bat`, `long` as first argument for the 32k mode):
```
llama-server -m MiMo-V2.6-Flash-MOPD-P160-OnPolicy-v3la-IQ2_XS.gguf -ngl 99 --jinja -c 16384 -fit off --expert-streaming ^
-ot "blk\.(1[4-9]|2[0-9]|3[0-3])\.ffn_(gate|up|down)_exps\.weight=CPU" ^
--temp 1.0 --top-p 0.95
```
**Smaller GPUs:** move more middle layers to the CPU by widening the `-ot` layer range (each middle layer holds
~1 GiB of experts; keep layers 0–11 and 36–47 on the GPU if you can, they are the most sensitive). Speed depends on
whether the CPU-side experts fit in RAM: up to ~20 GiB in RAM decode at 6–10 tok/s here; with 56 GiB paged from NVMe
(the full, unpruned model) it was 2.6 tok/s. The lock list (`hotlock_L14-33.txt`) only applies to layers 14–33;
regenerate it with `hot_lock_list.py` for other ranges or leave the variable unset.
The runtime patches and build steps are in `runtime/patches/` (llama.cpp `f5e85d43a` + Swap-MoE + ours). Stock
llama.cpp ignores `LLAMA_EXPERT_LOCK_LIST` and runs the model without that feature.
## Known limitations
- Pruning removes capacity: expect less world knowledge than the original (e.g. a wrong statute number in a German
legal answer) and occasional character-level typos in long German texts.
- First-shot code is mediocre; review it.
- Reversing a word letter by letter fails (all pruned variants).
- Experimental release; speed figures are from one machine.
## License
MIT, like the base model. The license text is provided in [LICENSE](LICENSE); sources and credits in [NOTICE.md](NOTICE.md).
---
中文说明 · 樱花 (Simplified Chinese)
> English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 [README_zh.md](README_zh.md)。
# Sakura — MiMo-V2.6-Flash-MOPD P160(专家剪枝,支持 SSD 流式加载)
Sakura MiMo P160 是 Xiaomi 的 MiMo-V2.6-Flash-MOPD 的一个实验性专家剪枝 GGUF:每个 MoE 层 256 个路由专家中保留 160 个,
约 53.79 GiB。它的设计目标是在单台 64 GB 的 AMD Strix Halo(Radeon 8060S,Vulkan)上运行,
约 35 GiB 放在 GPU 上,其余专家由系统内存计算,即下文介绍的 **MiMo SSD Streaming** 方案。重点:德语指令、长规划提示和编程。
这是独立的社区构建版本,**不是** Xiaomi 的官方发布。
属于 **[Sakura Micro](https://huggingface.co/collections/webmp3/sakura-micro-6aba74331f2e996ba1268a92)** 系列:
大模型的紧凑、可本地运行的衍生版本(系列:
[Sakura](https://huggingface.co/collections/webmp3/sakura-6aba73f6376c82b665cf2ea2),
[Sakura Mini](https://huggingface.co/collections/webmp3/sakura-mini-6aba73f6a5a41296a3540373),
[Sakura Micro](https://huggingface.co/collections/webmp3/sakura-micro-6aba74331f2e996ba1268a92))。
## 来源
- 基础模型:[XiaomiMiMo/MiMo-V2.6-Flash-MOPD](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD),即 MiMo-V2.6-Flash-RL 的 MOPD2
升级版(总计 309B / 激活 15B,48 层,256 个专家,top-8)。它的路由器与 RL 检查点逐位相同,因此在 RL 上测得的专家选择可以精确迁移。
- 由 Xiaomi 的原始权重构建:只下载了保留的专家(HTTP range),MXFP4 专家被无损重新打包,FP8 张量的转换方式与 llama.cpp 的转换器完全相同(已在 RL 发布版上与参照逐字节核对)。
- 专家选择(逐层):保护那些承载了完整模型在 **它自己的德语回答**(针对长编程/规划提示,on-policy 测量)上 80 % 路由量的专家,然后由解码轨迹中的热点专家、REAP 显著性分段和路由 token 占比来决定。仅在外来文本上调好的选择虽然困惑度更好,但在聊天中会循环;改为在模型自己的输出上测量后,这一问题得到解决。
- 量化:使用 llama.cpp 和重要性矩阵(Baekpica 的,已裁剪到保留的专家)。第 12–35 层的专家为 IQ2_XXS/IQ2_XS,其余层为 IQ2_XS/IQ2_S,第 44/46/47 层的 down 投影为 IQ3_XXS;注意力为 IQ4_XS,稠密 FFN 为 Q6_K,嵌入为 Q8_0,输出为 Q6_K。
## 发布文件
`MiMo-V2.6-Flash-MOPD-P160-OnPolicy-v3la-IQ2_XS.gguf`
- 量化:混合低比特,专家大部分为 IQ2_XS(详见上文);文件名中的 `IQ2_XS` 是占主导的类型
- 大小:57,761,339,136 字节(53.79 GiB)
- SHA-256:`90dbd32b5d41ec902da81d0c369a8649cb29c059d7d9624a0c2f49511fab3b1b`
- 参数量:剪枝后总计约 195B(原为 309B),每个 token 激活约 15B(不变)
- 可使用随附的 `SHA256SUMS` 文件校验下载。
## 测量结果
| 构建 | 大小 | PPL 德语 | PPL 代码 | 聊天检查 |
|---|---:|---:|---:|---|
| **Sakura MiMo P160(本次发布,MOPD)** | **53.8 GiB** | **6.02** | **5.80** | 短提示 7/8 · 德语压力测试 6/6 · 长文本 4/4 · 路线图 2/2(temperature 1.0)· 无循环 |
| 相同的选择和配方,RL 检查点 | 53.8 GiB | 6.02 | 5.87 | 路线图:temperature 0 时 1/2(反刍式循环),0.6 时 2/2 |
| 144 个专家,RL(由 TrevorJS IQ2 裁剪) | 53.5 GiB | 6.12 | 6.10 | 短提示 7/8 · 长文本 4/4 · 路线图 2/2(temperature 0) |
| 完整的 MiMo-V2.6-Flash-RL(TrevorJS IQ2,参照) | 91.9 GiB | 5.04 | 4.23 | — |
本次发布的解码速度:在桌面正常并行使用的情况下测得,16k 上下文时 **9–10.5 tok/s**,32k 时 5.6–7.7 tok/s(详见下文)。
困惑度:留出的德语和代码文本,8 × 512 个 token,未用于选择或校准(越低越好)。
速度:AMD Ryzen AI Max+ 395 / Radeon 8060S,64 GB,Vulkan,采用下文的 MiMo SSD Streaming 方案,在桌面正常并行使用(浏览器、聊天应用;空闲内存接近零)的情况下测得,上下文大小的影响见下面的速度表。
聊天检查:简短的事实/算术提示、长篇自由文本、六个德语中等长度回答,以及两个带固定结束语的德语长路线图/部署提示。这些只是小规模的功能性检查,不是关于相对于原始模型的整体质量的声明。
## 为什么是 160 个专家
我们在这台机器上,用同一个 MiMo-V2.6-Flash 系列构建并测试了几种专家数量
(留出的 PPL 德语 / 代码;越低越好):
| 每层保留的专家数 | 大小 | PPL 德语 | PPL 代码 | 聊天 |
|---|---:|---:|---:|---|
| 128(热点专家选择) | 48.0 GiB | 14.50 | 7.30 | 德语崩坏(词语杂烩、循环) |
| 128(保护德语) | 48.0 GiB | 5.97 | 6.99 | 德语正常,但代码循环且格式出错 |
| 144(on-policy) | 53.5 GiB | 6.12 | 6.10 | 稳定 |
| **160(on-policy,本次发布)** | **53.8 GiB** | **6.02** | **5.80** | 稳定,temperature 1.0 下无循环 |
| 256(完整模型,参照) | 91.9 GiB | 5.04 | 4.23 | —(此处 2.6 tok/s,大部分从 NVMe 换页) |
128/144 个专家的行以及完整模型参照都是 MiMo-V2.6-Flash-RL 的构建。相同的 160 专家选择在 RL 检查点上得分为 6.02 / 5.87,所以从 144 到 160 本身就有帮助;MOPD 在代码上又略有提升。
较小的 REAP/剪枝程度会造成明显的质量损失:在 128 个专家时,德语或代码会崩坏。在 64 GB 机器上仍能以可用速度运行的变体中,160 个专家的质量最好:该文件只比 144 专家的构建略大一点,因为中间层使用了稍小的量化类型。保留更多专家会把更多权重推出 GPU,代价是速度。**在我们测试过的变体中**,160 是 64 GB 的最佳平衡点;我们没有构建 176 及以上。
## 使用原版 llama.cpp 运行
任何支持 `mimo2` 的较新 llama.cpp 都能加载这个 GGUF。如果 GPU 上有足够内存放下整个文件:
```
llama-server -m MiMo-V2.6-Flash-MOPD-P160-OnPolicy-v3la-IQ2_XS.gguf -ngl 99 --jinja -c 32768 --temp 1.0 --top-p 0.95
```
请使用 temperature 1.0 / top-p 0.95(Xiaomi 的推荐):在我们的测试中,0–0.6 时,长思考任务有时会陷入反刍式循环;1.0 时则没有。
## MiMo SSD Streaming — 用更少的显存运行
该模型比 32 GB 的 GPU 更大。MiMo SSD Streaming 把注意力、离群值较多的早期层和后期层保留在 GPU 上,让 CPU 从内存中计算中间若干层(14–33,约 20 GiB)的专家,并以 NVMe 作为后备存储。各组件(都在本仓库的 `runtime/` 中):
| 组件 | 作用 |
|---|---|
| `-ot "…ffn_(gate\|up\|down)_exps\.weight=CPU"` | 将选定层的专家保留在主机内存(mmap)中;使模型适配约 35 GiB 的 GPU |
| Swap-MoE `--expert-streaming`([ek15072809/Swap-MoE](https://github.com/ek15072809/Swap-MoE))+ 我们的 GPU 防护补丁 | 按需从 NVMe 调入这些专家,而不是一开始就全部加载 |
| Op-offload **开启**(不使用 `--no-op-offload`) | 处理提示时将用到的专家复制到 GPU(专家在内存中时很快) |
| 专家锁(`LLAMA_EXPERT_LOCK_LIST`,我们的补丁,可选) | 把最常用的 CPU 侧专家固定在内存中(`VirtualLock`),使多 token 批处理不会触发缺页 |
| 聊天解析中的 UTF-8 清理(我们的补丁) | 剪枝后的模型有时会输出损坏的 emoji 字节序列;原版 llama-server 随后会以 HTTP 500 失败。现在回复会保留,损坏的字节变为 U+FFFD |
**在这台机器上的速度(解码,tok/s;桌面并行使用,空闲内存接近零):**
| 模式 | 上下文 | 锁 | 解码 | 预填充(4k token 提示) |
|---|---:|---|---:|---:|
| **fast(默认)** | 16k | 无 | **9–10.5** | 13–25 |
| long | 32k | 最热的 30 %(约 6 GiB) | 5.6–7.7 | 14–18 |
| (24k,仅供参考) | 24k | 无 | 6.6–7.4 | 32 |
所有行都使用默认的 `-ub 512`,它的整体速度最好。更大的 ubatch 用解码换预填充:
| `-ub` | 预填充(2k 提示) | 解码 |
|---:|---:|---:|
| **512(默认)** | 12.7 | **9.6** |
| 1024 | 22.3 | 5.9 |
| 2048 | 34–39(4k 提示) | ~5.3 |
当 CPU 侧的专家放不进内存时,每个 ubatch 会从 NVMe 读取一次,因此更大的 ubatch 会加快预填充;但它们更大的 GPU 缓冲区会溢出到共享系统内存,抢占这些专家所需的内存,使解码速度几乎减半。只有当提示长度超过回答长度约两倍时(例如一次性读取大文件),更大的 ubatch 才划算;对于带有长推理的编程,请保持 512。`-ub 4096` 在这台机器上耗尽了 GPU 内存。
上下文大小很重要,因为 KV 缓存和 GPU 缓冲区会溢出到共享系统内存;在 16k 时,仍有足够的内存留给 CPU 侧的专家。在内存充裕的安静机器上,同一个运行时在使用专家锁的情况下达到了 8.9 tok/s 解码、约 120 tok/s 预填充,以及 2/4/8 个 token 批处理时的 15/18/17 tok/s(用我们的 144 专家构建测得)。n-gram 推测解码(`--spec-type ngram-mod`)在返回编辑后的文件时对基于 RL 的构建有帮助,但对这个 MOPD 构建没有帮助(接受率约 50 %),因此默认关闭。
64 GB Strix Halo 的示例(`runtime/start_server_example.bat`,第一个参数为 `long` 时使用 32k 模式):
```
llama-server -m MiMo-V2.6-Flash-MOPD-P160-OnPolicy-v3la-IQ2_XS.gguf -ngl 99 --jinja -c 16384 -fit off --expert-streaming ^
-ot "blk\.(1[4-9]|2[0-9]|3[0-3])\.ffn_(gate|up|down)_exps\.weight=CPU" ^
--temp 1.0 --top-p 0.95
```
**较小的 GPU:** 通过加宽 `-ot` 的层范围,将更多中间层移到 CPU(每个中间层约含 1 GiB 的专家;如果可以,请把 0–11 层和 36–47 层保留在 GPU 上,它们最敏感)。速度取决于 CPU 侧专家能否放进内存:最多约 20 GiB 放在内存中时,这里解码为 6–10 tok/s;当 56 GiB 从 NVMe 换页时(完整的未剪枝模型)为 2.6 tok/s。锁列表(`hotlock_L14-33.txt`)只适用于 14–33 层;对于其他范围,请用 `hot_lock_list.py` 重新生成,或者不设置该变量。
运行时补丁和构建步骤在 `runtime/patches/` 中(llama.cpp `f5e85d43a` + Swap-MoE + 我们的补丁)。原版 llama.cpp 会忽略 `LLAMA_EXPERT_LOCK_LIST`,在没有该功能的情况下运行模型。
## 已知局限
- 剪枝会去除容量:预计世界知识会少于原始模型(例如在一个德语法律回答中写错法条编号),并且在长德语文本中偶尔出现字符级的拼写错误。
- 第一次生成的代码质量一般;请自行审查。
- 逐字母反转一个单词会失败(所有剪枝变体都是如此)。
- 实验性发布;速度数据来自一台机器。
## 许可证
MIT,与基础模型相同。许可证文本见 [LICENSE](LICENSE);来源和致谢见 [NOTICE.md](NOTICE.md)。