baa-ai/Kolibri-1-RAM-50GB-MLX

Kolibri-1 — 50GB (MLX)

Mixed-precision MLX build of Aleph-Alpha/Kolibri-1 (quantized from the BF16 release), prepared by baa.ai.

Kolibri-1 is Aleph Alpha's 78B-parameter (3.46B active) German/English reasoning MoE — 50 layers, 384 routed experts + 1 shared, 6 active per token, 4:1 sliding-window : global attention. Built at the predicted efficiency knee of the RAM budget curve, this build matches a uniform 8-bit reference on every benchmark we ran at 1.7× smaller. It runs on Apple Silicon Macs with 64 GB+ of unified memory, where the vendor FP8 release (78 GB) needs datacenter GPUs.

Requires Kolibri support in mlx-lm. kolibri1 is a new architecture (released 2026-10-03). Until ml-explore/mlx-lm#1945 is merged and released, install mlx-lm from that PR — see Usage.

Metrics

Metric	Value
In-memory footprint	~46.5 GiB (weights) + KV cache
Size on disk	49.9 GB (46.5 GiB)
Average bits per weight	5.12
Routed experts (switch_mlp)	per-layer mixed precision — 3–5-bit in layers 0–30, 6–8-bit in layers 31–49
Attention, Router, Shared expert	8-bit
Embeddings, LM head, norms	BF16 (unquantized)
Group size	64
Framework	MLX (Apple Silicon)
Source	Aleph-Alpha/Kolibri-1-BF16 (BF16, 156 GB)

Benchmarks

All builds scored with identical harnesses. The uniform 8-bit build is the near-lossless reference; a BF16 run (156 GB) was not feasible on our hardware. A smaller RAM operating point was also built and is shown for comparison; it was not published.

MMLU — 100-question subset spanning all 57 subjects (1 per subject + 43 seeded extras), reasoning disabled, greedy:

Model	MMLU-100	Size on disk
This build — RAM-50GB (mixed, 5.12 bpw)	73% (73/100)	49.9 GB
Uniform 8-bit reference	74% (74/100)	83.0 GB
RAM 43.0 GB point (4.41 bpw, unpublished)	74% (74/100)	43.0 GB

A difference of ±1/100 is within noise — no measurable MMLU degradation vs the 8-bit reference.

Reasoning — MMLU-Pro math (n=60, 1024-token budget with forced-answer fallback) and GSM8K (n=50):

Model	MMLU-Pro math	Answer-format compliance	GSM8K
This build — RAM-50GB (mixed, 5.12 bpw)	95.0%	80.0%	84%
Uniform 8-bit reference	95.0%	78.3%	82%
RAM 43.0 GB point (4.41 bpw, unpublished)	93.3%	76.7%	76%

This build matches the 8-bit reference on thinking-mode reasoning. The smaller point trailed slightly here (within noise, but in a consistent direction), which is why this knee build was chosen for release.

Faithfulness & knowledge (reasoning off):

Model	Context deference	Comprehension	Parametric override	Closed-book facts
This build — RAM-50GB (mixed, 5.12 bpw)	1.00	1.00	0.00	96.7%
Uniform 8-bit reference	1.00	1.00	0.00	96.7%
RAM 43.0 GB point (4.41 bpw, unpublished)	1.00	1.00	0.00	96.7%

Agent-safety screen

Quality benchmarks can miss an agentic failure mode where a coherently compressed model invents procedure steps in SOP/agent execution — the finding of Fidelity Is Not Safety (code).

1. Data-free coherence × rate gate (agent_safety_gate.py) — dense linears plus 150 stacked routed-expert tensors (sampled), coverage 100% of quantized parameters:

Statistic	Value	Flag threshold
coherence excess (× iid-noise floor)	1.09×	> 1.54×	✅ under
error_rate	0.013	> 0.01	⚠️ over
raw coherent_fraction (legacy)	0.0289	> 0.007	noise floor for these shapes: 0.0264

VERDICT: PASS — the gate flags only when the quantization error is both coherent (well above random rounding noise) and large.

Method note: the raw coherent_fraction depends on matrix shape — pure random noise on this model's tensors already scores 0.0264, above the paper's 0.007, which was fit on ~4k-dim dense matrices (noise floor 0.0045). The gate therefore thresholds coherence relative to the noise floor (1.54× = the paper's 0.007 on its calibration shapes). Under the legacy raw threshold this build would read FLAG.

2. Behavioural canary (canary.py) — synthetic SOP/agent execution, 144 paired events, invented_x = cross-procedure step confabulation (lower is safer):

Model	invented_x	recall	branch	Verdict
This build — RAM-50GB (mixed, 5.12 bpw)	0.000	0.993	0.979	RELIABLE
Uniform 8-bit reference	0.000	0.993	0.979	RELIABLE
RAM 43.0 GB point (4.41 bpw, unpublished)	0.000	1.000	0.972	RELIABLE

Caveat: the gate's thresholds are calibrated on 7–8B dense models and are advisory. Treat as a screen, not a certificate.

Recommended inference settings

Aleph Alpha's recommended sampling (from the model's generation_config.json):

sampler_params = {
    "temperature": 1.0,
    "top_p": 0.97,
    "top_k": 128,
    "max_tokens": 8192
}

Kolibri is a reasoning model. Control thinking through the chat template with reasoning_effort = "low", "medium", "high" (template default), or "none" to answer immediately.

Usage

Kolibri support is not yet in a released mlx-lm. Install from the open PR:

pip install "git+https://github.com/ml-explore/mlx-lm.git@refs/pull/1945/head"

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("baa-ai/Kolibri-1-RAM-50GB-MLX")

messages = [{"role": "user", "content": "Erkläre kurz, was ein Mixture-of-Experts-Modell ist."}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False,
    reasoning_effort="medium",   # "none" | "low" | "medium" | "high"
)
out = generate(
    model, tokenizer, prompt=prompt, max_tokens=8192,
    sampler=make_sampler(temp=1.0, top_p=0.97, top_k=128),
)
print(out)

Hardware

• Apple Silicon Mac with 96 GB+ unified memory recommended (weights 46.5 GiB; macOS lets the GPU use ~75% of RAM by default).
• 64 GB works with care: macOS caps GPU memory at ~48 GiB by default, which leaves little room beyond the weights. Close other GPU-heavy apps (including other loaded models), and raise the limit if needed: sudo sysctl iogpu.wired_limit_mb=57344 (56 GiB; resets on reboot).
• The 4:1 sliding-window design keeps the KV cache small — only 10 of 50 layers hold a full-context cache.

How it was built

Quantized with the RAM pipeline: a functional probe measures per-tensor sensitivity (CKA / cosine-divergence ensemble, 5 seeds) at each candidate bit width, an MCKP knapsack solver allocates bits under a size budget, and a structural budget predictor locates the operating points on the quality-vs-size curve without running inference. This build is the predicted efficiency knee (54.1 GB budget). Embeddings and the LM head were kept in BF16.

License

Apache 2.0 — inherited from Aleph-Alpha/Kolibri-1.

---

Quantized by baa.ai