Instructions to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS
# Run inference directly in the terminal:
llama cli -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS
# Run inference directly in the terminal:
llama cli -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS
# Run inference directly in the terminal:
./llama-cli -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Use Docker

docker model run hf.co/webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Llama new
LM Studio
Jan

vLLM

How to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Ollama
How to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with Ollama:
```
ollama run hf.co/webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS
```
Unsloth Desktop

How to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with Docker Model Runner:
```
docker model run hf.co/webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS
```

Lemonade

How to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Run and chat with the model

lemonade run user.Sakura-MicroQuality-Kolibri-1-365E-GGUF-IQ2_XS

List all available models

lemonade list

Hermes Agent

How to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF:IQ2_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Sakura MicroQuality — Kolibri-1 365 experts (GGUF, three sizes)

Made in Germany: Kolibri-1 is the open 78B model of the German company Aleph Alpha, a mixture-of-experts model (3.5B active parameters, 384 routed experts plus one shared expert per layer) for German and English, with a thinking mode and tool calling. This repository holds the same files as Sakura-MicroQuality-Kolibri-1-GGUF with the 19 least used routed experts of every layer removed (365 of 384), about 4.5 % smaller. This repository holds three GGUF files of different size made from the Q8_0 GGUF of the original model (near-lossless; all quality numbers use it as the reference) with our measured mixed-codec method. Pick one on the Files page; the table below says what each file is. Independent community quantization, not an official Aleph Alpha release. Part of the Sakura Micro line.

Sakura MicroQuality

Smaller than the standard quant of the same class, and much closer to the original. Every file here is measured against the standard llama.cpp quant of its size class (same importance matrix, same texts, same machine) and only files that beat it are published. Measured as KL divergence to the Q8_0 source on 4 held-out texts (en, dev, wiki, de); lower is better.

File	Size	Standard quant of its class	KL divergence of this file	Better on
`Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf`	19.99 GiB	IQ2_XS (21.41 GiB)	58 % lower	4 of 4 texts
`Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf`	27.45 GiB	IQ3_XXS (28.12 GiB)	50 % lower	4 of 4 texts
`Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf`	36.31 GiB	IQ4_XS (38.91 GiB)	39 % lower	4 of 4 texts

Needs a patched llama.cpp for now. The kolibri1 architecture is not in mainline llama.cpp yet (feature request #29922). Apply the patch of Hob-forge/Kolibri-1-GGUF to llama.cpp commit 836d571, the same as for the other community GGUF files of this model. This card will be updated when upstream support lands.

Quality at a glance

Same method, same texts, same machine for every file; lower KL divergence means closer to the near-lossless Q8_0 reference:

Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf vs llama.cpp default IQ2_XS (21.41 GiB): KL divergence to the near-lossless Q8_0 reference 58 % lower on average over all 4 texts (1.41 GiB smaller).
Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf vs llama.cpp default IQ3_XXS (28.12 GiB): KL divergence to the near-lossless Q8_0 reference 50 % lower on average over all 4 texts (0.67 GiB smaller).
Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf vs llama.cpp default IQ4_XS (38.91 GiB): KL divergence to the near-lossless Q8_0 reference 39 % lower on average over all 4 texts (2.60 GiB smaller).

Why: the bit budget per tensor is allocated from measured sensitivity (see How it was made), not from fixed rules. Full table below.

Which file should I take?

File	For
`Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf` (19.99 GiB, 2.31 bpw)	smallest: for tight memory budgets; expect visible quality loss
`Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf` (27.44 GiB, 3.17 bpw)	middle: best balance of size and quality for most machines
`Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf` (36.30 GiB, 4.19 bpw)	largest: closest to the Q8_0 reference of the three

The files

File	Size	bits/weight	KLD en	KLD dev	KLD wiki	KLD de	same top token (avg)	Tensor types by size
`Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf`	19.99 GiB	2.31	0.0758	0.0387	0.2285	0.1025	90.5 %	IQ2_XXS 70%, IQ2_XS 11%, IQ3_XXS 8%, Q8_0 5%, IQ4_XS 4%, Q2_K 2%
`Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf`	27.44 GiB	3.17	0.0452	0.0266	0.1346	0.0578	93.5 %	IQ3_XXS 68%, Q8_0 8%, IQ2_XS 5%, IQ2_S 5%, IQ4_XS 4%, IQ3_S 3%, Q2_K 2%, IQ2_XXS 2%
`Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf`	36.30 GiB	4.19	0.0305	0.0156	0.0938	0.0409	94.7 %	IQ4_XS 65%, IQ3_S 14%, Q4_K 8%, Q8_0 7%, IQ3_XXS 4%

"Tensor types by size" lists the share of the file's bytes per ggml type (all are standard llama.cpp types, so the files run in stock llama.cpp). Quality columns: mean KL divergence of the quantized model's next-token distribution against the Q8_0 source on held-out text (lower is better) and how often the most likely token is unchanged (higher is better).

Comparison at similar size

Same texts, same context, same chunks, measured by us with llama-perplexity (near-lossless Q8_0 source = reference; PPL of the Q8_0 reference: en: 2.460, dev: 1.928, wiki: 24.848, de: 2.908). The reference quants are: llama.cpp default recipe, same importance matrix (ours), measured the same way. Sorted by size.

Quant	Source	Size	KLD en	KLD dev	KLD wiki	KLD de	PPL en	same top token (avg)
`Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf`	this repository	19.99 GiB	0.0758	0.0387	0.2285	0.1025	2.469	90.5 %
IQ2_XS	llama.cpp default recipe, same importance matrix (ours)	21.41 GiB	0.1859	0.1113	0.4540	0.2548	2.653	84.6 %
`Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf`	this repository	27.45 GiB	0.0452	0.0266	0.1346	0.0578	2.452	93.5 %
IQ3_XXS	llama.cpp default recipe, same importance matrix (ours)	28.12 GiB	0.0917	0.0519	0.2667	0.1187	2.453	89.6 %
`Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf`	this repository	36.31 GiB	0.0305	0.0156	0.0938	0.0409	2.460	94.7 %
IQ4_XS	llama.cpp default recipe, same importance matrix (ours)	38.91 GiB	0.0566	0.0253	0.1550	0.0616	2.514	93.2 %

Method: 12 chunks of 512 tokens per text; texts: English held-out text, developer-text held-out set, English encyclopedia held-out text (not used in any calibration), German held-out text (not used in any calibration). Single measurement on one machine; small KLD differences at similar size are not a quality ranking.

How it was made

The source is the Q8_0 GGUF from Hob-forge/Kolibri-1-GGUF (converted from Aleph Alpha's FP8 checkpoint; near-lossless). We did not convert from the 156 GB BF16 weights, so the reference for every number on this card is that Q8_0. The importance matrix was computed by us (llama-imatrix) on German and English text, code and the model's own chat answers with thinking.
Every large weight matrix was quantized once per candidate type (Q2_K, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, Q3_K, IQ4_XS, Q4_K, Q8_0) with llama-quantize and the importance matrix; the error of each choice was estimated per matrix (importance-weighted) and scaled by a sensitivity factor measured with real KL divergence: every group of tensors (for example the FFN down projections, the attention value projections, the first and last layers) was moved alone to a lower-bit type, and the KL divergence it caused was compared with its quantization error.
An exact budget allocation (multiple-choice knapsack over bytes) picked one type per matrix for each target size; the final model was assembled from the stored tensors without re-quantizing. We do not publish per-tensor choices. Norms and small tensors stay at high precision.

Limits

Measured only with KL divergence and perplexity on short held-out texts and not with downstream benchmarks; do not read it as a task-quality claim.
Quantization always costs some quality; the smallest file costs the most.
Requires a patched llama.cpp until upstream support lands (see the banner at the top). All numbers on this card were measured with that build (Vulkan backend).
What was removed: in every layer the 19 routed experts that our calibration texts (German and English text, code, and the model's own chat answers with thinking) selected least often. On those texts only about 0.07 % of all expert selections fell on the removed experts (at most 0.23 % in any layer); 375 of the 950 removed experts were never selected at all. The cut slices the already quantized expert tensors, router weights and router bias, with no requantization and no retraining; top-6 routing is unchanged. The files are byte-for-byte slices of the matching files in Sakura-MicroQuality-Kolibri-1-GGUF. Content that our texts do not cover may rely on removed experts, so check your own use case.
Requantizing from a Q8_0 source adds a very small extra error compared with converting from BF16; the quality columns compare every file against that same Q8_0 source of the full model.
Recommended settings from the model card: temperature 1.0, top-p 0.97, top-k 128; reasoning effort is set per request (none, low, medium, high).

Credits

Original model: Kolibri-1 (Aleph Alpha, Apache-2.0), Apache-2.0 license (included as LICENSE).
Q8_0 source and the llama.cpp patch: Hob-forge/Kolibri-1-GGUF (Hob-forge / Seraphiel102). Reference quants: llama-quantize default recipes of the full model with our importance matrix.
GGUF format and tools: ggml-org/llama.cpp (MIT).

中文说明 · 樱花 (Simplified Chinese)

English above. 本节为上文的中文翻译(Sakura = 樱花 yīnghuā);完整的独立中文版见 README_zh.md。

Sakura MicroQuality — Kolibri-1 365 位专家(GGUF,三种大小)

德国制造: Kolibri-1 是德国公司 Aleph Alpha 的开源 78B 模型,为混合专家(MoE)模型(3.5B 激活参数,每层 384 个路由专家加 1 个共享专家),面向德语和英语,带有思考模式和工具调用。本仓库包含与 Sakura-MicroQuality-Kolibri-1-GGUF 相同的文件,但每层移除了使用最少的 19 个路由专家(384 个中保留 365 个),大小约小 4.5 %。本仓库包含 三个不同大小的 GGUF 文件,它们由原模型的 Q8_0 GGUF(近乎无损;所有质量数据均以它为参照)通过我们实测得出的混合编码方法制作。请在 Files 页面选择其中一个;下表说明了每个文件的用途。这是独立的社区量化版本,不是 Aleph Alpha 的官方发布。属于 Sakura Micro 系列。

Sakura MicroQuality

比同级别的标准量化更小,并且更接近原始模型。 这里的每个文件都与其大小级别的标准 llama.cpp 量化(相同的重要性矩阵、相同的文本、同一台机器)进行了对比测量,只有优于它的文件才会发布。测量方式为在 4 个留出文本(en、dev、wiki、de)上相对于 Q8_0 源文件的 KL 散度;越低越好。

文件	大小	同级别的标准量化	此文件的 KL 散度	更优的文本数
`Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf`	19.99 GiB	IQ2_XS (21.41 GiB)	低 58 %	4 个中的 4 个
`Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf`	27.45 GiB	IQ3_XXS (28.12 GiB)	低 50 %	4 个中的 4 个
`Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf`	36.31 GiB	IQ4_XS (38.91 GiB)	低 39 %	4 个中的 4 个

目前需要打过补丁的 llama.cpp。 kolibri1 架构尚未进入 llama.cpp 主线(功能请求 #29922)。请将 Hob-forge/Kolibri-1-GGUF 的补丁应用到 llama.cpp 提交 836d571 上,与该模型的其他社区 GGUF 文件相同。上游支持落地后,本卡片会更新。

质量概览

每个文件使用相同的方法、相同的文本、同一台机器;KL 散度越低,越接近近乎无损的 Q8_0 参照:

Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf 对比 llama.cpp 默认的 IQ2_XS (21.41 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 58 %(小 1.41 GiB)。
Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf 对比 llama.cpp 默认的 IQ3_XXS (28.12 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 50 %(小 0.67 GiB)。
Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf 对比 llama.cpp 默认的 IQ4_XS (38.91 GiB):相对于近乎无损的 Q8_0 参照,在全部 4 个文本上平均 KL 散度 低 39 %(小 2.60 GiB)。

原因:每个张量的比特预算是根据实测的敏感度分配的(见 制作方式),而不是按固定规则。完整表格见下文。

我该选哪个文件?

文件	适用场景
`Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf`(19.99 GiB,2.31 bpw)	最小:适合内存预算紧张的情况;预计质量会有明显损失
`Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf`(27.44 GiB,3.17 bpw)	居中:对大多数机器而言,大小与质量的最佳平衡
`Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf`(36.30 GiB,4.19 bpw)	最大:三者中最接近 Q8_0 参照

文件

文件	大小	bits/weight	KLD en	KLD dev	KLD wiki	KLD de	same top token (avg)	各张量类型(按大小)
`Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf`	19.99 GiB	2.31	0.0758	0.0387	0.2285	0.1025	90.5 %	IQ2_XXS 70%, IQ2_XS 11%, IQ3_XXS 8%, Q8_0 5%, IQ4_XS 4%, Q2_K 2%
`Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf`	27.44 GiB	3.17	0.0452	0.0266	0.1346	0.0578	93.5 %	IQ3_XXS 68%, Q8_0 8%, IQ2_XS 5%, IQ2_S 5%, IQ4_XS 4%, IQ3_S 3%, Q2_K 2%, IQ2_XXS 2%
`Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf`	36.30 GiB	4.19	0.0305	0.0156	0.0938	0.0409	94.7 %	IQ4_XS 65%, IQ3_S 14%, Q4_K 8%, Q8_0 7%, IQ3_XXS 4%

“各张量类型(按大小)”列出每个 ggml 类型占文件字节数的比例(全部是标准的 llama.cpp 类型,因此这些文件可在原版 llama.cpp 中运行)。质量列:量化模型的下一个 token 分布相对于 Q8_0 源文件在留出文本上的平均 KL 散度(越低越好),以及最可能的 token 保持不变的频率(越高越好)。

相近大小的比较

相同的文本、相同的上下文、相同的分块,由我们使用 llama-perplexity 测量(近乎无损的 Q8_0 源文件 = 参照;Q8_0 参照的 PPL:en: 2.460,dev: 1.928,wiki: 24.848,de: 2.908)。参照量化为:llama.cpp 默认配方、相同的重要性矩阵(我们的)、以相同方式测量。按大小排序。

量化	来源	大小	KLD en	KLD dev	KLD wiki	KLD de	PPL en	same top token (avg)
`Sakura-Kolibri-1-365E-IQ2_XS-19.99GiB.gguf`	本仓库	19.99 GiB	0.0758	0.0387	0.2285	0.1025	2.469	90.5 %
IQ2_XS	llama.cpp 默认配方,相同的重要性矩阵(我们的)	21.41 GiB	0.1859	0.1113	0.4540	0.2548	2.653	84.6 %
IQ3_XXS	llama.cpp 默认配方,相同的重要性矩阵(我们的)	28.12 GiB	0.0917	0.0519	0.2667	0.1187	2.453	89.6 %
`Sakura-Kolibri-1-365E-IQ3_XXS-27.45GiB.gguf`	本仓库	27.45 GiB	0.0452	0.0266	0.1346	0.0578	2.452	93.5 %
`Sakura-Kolibri-1-365E-IQ4_XS-36.31GiB.gguf`	本仓库	36.31 GiB	0.0305	0.0156	0.0938	0.0409	2.460	94.7 %
IQ4_XS	llama.cpp 默认配方,相同的重要性矩阵(我们的)	38.91 GiB	0.0566	0.0253	0.1550	0.0616	2.514	93.2 %

方法:每个文本取 12 个块,每块 512 个 token;文本为:英文留出文本、开发者文本留出集、英文百科留出文本(未用于任何校准)、德文留出文本(未用于任何校准)。这是在一台机器上的单次测量;相近大小下 KLD 的细小差异并不构成质量排名。

制作方式

来源是 Hob-forge/Kolibri-1-GGUF 的 Q8_0 GGUF(由 Aleph Alpha 的 FP8 检查点转换而来;近乎无损)。我们没有从 156 GB 的 BF16 权重转换,因此本卡片上每个数据的参照都是该 Q8_0。重要性矩阵由我们自己(llama-imatrix)在德语和英语文本、代码以及模型自己带思考的聊天回答上计算。
每个大型权重矩阵针对每种候选类型(Q2_K、IQ2_XXS、IQ2_XS、IQ2_S、IQ3_XXS、IQ3_S、Q3_K、IQ4_XS、Q4_K、Q8_0)使用 llama-quantize 和重要性矩阵各量化一次;每种选择的误差按矩阵估计(按重要性加权),并乘以一个 用真实 KL 散度测得的敏感度系数:每组张量(例如 FFN down 投影、注意力 value 投影、首尾几层)被单独移到较低比特的类型,并将它造成的 KL 散度与其量化误差进行比较。
精确的预算分配(以字节为单位的多选背包问题)为每个目标大小为每个矩阵挑选一种类型;最终模型由已存储的张量组装而成,无需重新量化。我们不公布逐张量的选择。归一化层和小张量保持高精度。

局限

仅用短的留出文本上的 KL 散度和困惑度测量,没有使用下游基准测试;请勿将其理解为任务质量的声明。
量化总会损失一些质量;最小的文件损失最大。
在上游支持落地之前,需要打过补丁的 llama.cpp(见顶部横幅)。本卡片上的所有数据都是用该构建(Vulkan 后端)测得的。
移除了什么: 在每一层中,移除被我们的校准文本(德语和英语文本、代码,以及模型自己带思考的聊天回答)选中次数最少的 19 个路由专家。在这些文本上,只有约 0.07 % 的专家选择落在被移除的专家上(任何一层最多 0.23 %);被移除的 950 个专家中有 375 个从未被选中。裁剪是对已量化的专家张量、路由器权重和路由器偏置进行切片,没有重新量化,也没有再训练;top-6 路由不变。这些文件是 Sakura-MicroQuality-Kolibri-1-GGUF 中对应文件的逐字节切片。我们的文本未覆盖的内容可能依赖被移除的专家,请检查你自己的使用场景。
与从 BF16 转换相比,从 Q8_0 源文件重新量化会增加极小的额外误差;质量列将每个文件与完整模型的同一个 Q8_0 源文件进行比较。
模型卡片推荐的设置:temperature 1.0、top-p 0.97、top-k 128;推理强度按请求设置(none、low、medium、high)。

致谢

原始模型:Kolibri-1 (Aleph Alpha, Apache-2.0),Apache-2.0 许可证(以 LICENSE 随附)。
Q8_0 源文件和 llama.cpp 补丁:Hob-forge/Kolibri-1-GGUF(Hob-forge / Seraphiel102)。参照量化:完整模型的 llama-quantize 默认配方,配以我们的重要性矩阵。
GGUF 格式和工具:ggml-org/llama.cpp (MIT)。