logic65/Qwen3.8-Whittle-MoE-27B-A17.8B-GGUF

Whittle MoE 27B (A18B) v2.2.1 GGUF

☕ Support this work

Whittle is built by one person on a grocery budget and rented GPU hours. If this research is useful to you, or you want to see it finished: ko-fi.com/davida81328. Every hour of GPU time goes straight into the next checkpoint, and every checkpoint, table and log lands in these repos.

Quantized builds of Whittle MoE 27B (A18B), a post hoc mixture of experts carved from Qwen3.8-27B and taught when to stop talking. v2.1 passed every release bar: 8 percent single turn loop rate (down from 69 at first release), 22 percent structured (down from 75), zero truncated answers, knowledge battery 28 of 39. Method, measurements, and the failures along the way are on the main model card.

Status

Earlier line (Aug 2026); the current Whittle models are the Whittle-Next / Whittle-Qwen-3.8 line: Whittle-Qwen-3.8-35B-A3B.

All tiers here are rebuilt from v2.2.1 (29 Aug 2026). The v2.2 tiers they replace, and the v2.1 tiers before them, are in this repo's git history (see Which file).

Which file

The files in this repo are Whittle-MoE-27B-A18B-v2.2.1-<tier>.gguf, plus mmproj-qwen38.gguf (optional vision projector, see below) and loop_test.py.

file	size	notes
Q8_0	28.7 GB	reference quality
Q6_K	23.1 GB
DQ5_K_XL	21.7 GB	DQ recipe: q8_0 embeddings/head, q6_k attention and shared expert
DQ4_K_XL	20.1 GB	DQ recipe
Q4_K_M	17.4 GB	stock allocation
DQ3_K_XL	14.7 GB	small, more attention bits than Q3_K_M
Q3_K_M	13.9 GB	new — small, stock allocation, no DQ recipe

Q8_0, Q6_K, Q4_K_M, DQ5_K_XL and DQ4_K_XL reproduce the v2.2 tensor allocation exactly, tensor for tensor. DQ3_K_XL does not: on this llama.cpp the 64 expert down-projections fall back to q5_1 where the v2.2 build emitted q5_0, which is chosen by the quantiser for 192-wide rows and is not settable by a flag. It is a q3_k_l build rather than a byte-match of the old file. Q3_K_M is new, added so there is a small tier without the DQ recipe.

The shared_expert_gate is stored at f32 in every tier, so the fix is carried at full precision regardless of the quantisation level. The gate tensor is 1-D, so llama-quantize keeps it at F32 in every tier automatically - the stop signal survives even DQ3.

v2.2 build sizes as published (binary GiB), with the "fits" guidance. The v2.2 tiers were replaced in place by the v2.2.1 builds above; the table above gives decimal GB as the Hub lists the files, this one gives binary GiB (for example the Q8_0 is 26.71 GiB = 28.7 GB), so the two differ by unit rather than by content, except DQ3_K_XL, which was rebuilt (see above).

file	size	fits	notes
Q8_0	26.7 GB	32 GB, or 24 GB with expert offload	best quality in practice
Q6_K	22.1 GB	24 GB	flat quant
DQ5_K_XL	20.2 GB	24 GB	dynamic, close to Q6 quality
DQ4_K_XL	18.8 GB	20 to 24 GB	dynamic, the size and quality sweet spot
Q4_K_M	16.2 GB	16 to 20 GB	flat quant, smallest standard build
DQ3_K_XL	13.9 GB	15 to 16 GB, no offload needed	dynamic, smallest usable tier
mmproj-qwen38.gguf	0.9 GB	adds about 1 GB	optional vision projector, see below

The BF16 tier is retired, and the v2.1 tiers were removed from the tree to keep one clear current version - pin revision d4b45cea of this repo if you need a v2.1 file (v2.1 is also git revision 3ae1efa7 of the model repo).

Measured

v2.2.1 (29 Aug 2026): the reasoning abort

Served with reasoning enabled, v2.2 could end its turn inside its own chain of thought instead of answering: its stop gate had been trained on sequences where the chat template opened a <think> block that the training data never closed, so the most heavily weighted signal in the set landed on a terminator inside the thinking block.

v2.2.1 replaces the 64 shared_expert_gate tensors — 0.66 MB, nothing else changed — with a gate trained on real chain-of-thought traces from the parent, where the block is closed by the teacher's own words. On a 24-prompt structural probe with reasoning on, turns that ended without ever emitting </think> fell from 21/24 to 3/24. Zeroing the gate entirely scores 11/24, so the new gate is doing real work rather than merely undoing the damage.

What this does not fix. Measured like-for-like against v2.2 with reasoning off on 36 held-out counted prompts: turns that stop cleanly 32/36 → 34/36, distinct-item exact counts 20/36 → 24/36, mean overrun 1.69× → 1.14×. Mean repeated-4-gram is unchanged at 0.141, still above our 0.05 target. Long lists of 45+ items remain weak (4/12 → 5/12). Take v2.2.1 for the reasoning abort, not for the counting.

v2.2 (28 Aug 2026): the stop gate

The 64 per-layer shared_expert_gate tensors carry a trained stop signal (0.33M params changed, nothing else). Measured against v2.1 on held-out prompts: list termination 18/36 -> 27/36, worst-window list repetition 0.237 -> 0.048, conversation turns terminating 27/48 -> 41/48. One honest regression: exact item counts dropped (8/36 -> 2/36) - it ends lists reliably but not at the requested count; pin the v2.1 files (git history of this repo) if exact-N lists are your use case. Full numbers and the root-cause story are on the model card.

What the DQ tiers are

A dynamic mix rather than a flat quant. The routed experts carry the tier's nominal bit width, while the parts that suffer most under compression keep more bits:

• The router stays at F16 in every DQ tier. It is tiny, around 20M parameters, but it decides which 16 of 64 expert slivers fire for each token. This model is its routing: it was brought back from gibberish by training the routers with every expert frozen, so they are the last thing to compromise.
• Shared experts and attention sit one or two tiers above the base type.
• Embeddings and the output head are protected in the upper tiers. In DQ3 they are deliberately dropped, because with a 248k vocabulary they alone were adding about 2.7 GB and pushing a supposed Q3 build above Q4_K_M.

One quirk worth knowing if you quantize this architecture yourself: the routed expert down projections have 192 wide rows, and k-quants need rows divisible by 256, so llama.cpp silently falls back to legacy q5_1 for them at 6 bits per weight no matter which tier you ask for. That is why a naive Q3 build of this model lands heavier than Q4_K_M. The DQ3 build accounts for it.

Naming note, because it matters: DQ is our own recipe, a fixed protection rule applied at quantize time. It is not Unsloth's UD, which uses closed source imatrix calibration to choose per layer bit widths from data. Same underlying idea, less sophisticated, and named differently on purpose.

Run it

llama-server -m Whittle-MoE-27B-A18B-v2.2-DQ4_K_XL.gguf -ngl 99 -c 8192 -fa on --jinja

The current file names carry v2.2.1 (for example Whittle-MoE-27B-A18B-v2.2.1-DQ4_K_XL.gguf); substitute the file you downloaded.

Any recent llama.cpp build with Qwen3.5 MoE support works. No fork, no patches.

Short on VRAM? Keep the routed experts in system RAM instead of shrinking the model:

llama-server -m <file>.gguf -ngl 99 --n-cpu-moe 32 -c 8192 -fa on --jinja

--n-cpu-moe N moves the experts of N layers (of 64) out to RAM while attention, the GDN layers, the routers and the shared experts stay on the GPU. Raise N until it fits: 64 puts every routed expert in RAM and needs only about 5 GB of VRAM. Each token touches just 16 of 64 slivers, so it costs speed, not sanity. Older builds without that flag can use -ot '\.ffn_(gate|up|down)_exps\.=CPU'.

Also on Ollama:

ollama run Whittle/Qwen3.8-Whittle-MoE-27B-A17.8B

Vision (experimental)

The parent is a vision language model and this MoE kept the whole multimodal interface intact through the carve. mmproj-qwen38.gguf is the parent's vision tower converted for llama.cpp:

llama-server -m <file>.gguf --mmproj mmproj-qwen38.gguf -ngl 99 -c 8192 -fa on --jinja

Then attach images in the built in web UI or through the API. Honest status: the routers were healed on text only and vision has not been through any measured gate. First eyes on tests show detailed, structured image description including small texture details and composition judgments. Treat it as a research bonus until numbers land.

Check the numbers yourself

The harness that produced the loop rates on the model card is in this repo:

curl -sLO https://huggingface.co/logic65/Qwen3.8-Whittle-MoE-27B-A17.8B-GGUF/resolve/main/loop_test.py
python3 loop_test.py http://localhost:8080
# ollama:  python3 loop_test.py http://localhost:11434 Whittle/Qwen3.8-Whittle-MoE-27B-A17.8B
# routers: pass the model name as the second argument or the server returns 400

Same prompts, thresholds, seeds and samplers used to gate every release. It runs about 90 generations, prints single turn, structured, multi turn and late turn failure rates, and writes every generation's scores to loop_test_results.json so you can inspect individual failures instead of trusting a summary. Standard library only.

A faster sibling: Whittle-Next-27B-A3B

This model keeps 17.8B parameters active per token: it is the stronger and slower of the two.

If you want the same 27B total at a fraction of the compute, Whittle-Next-27B-A3B (GGUF) activates only ~3B parameters per token, so it generates at the speed of a 3B model, and adds hyper-connection residual streams plus a 2B-row hashed n-gram memory carried over from Qwen3.8-Flash-Next. It runs on stock llama.cpp with no patches.

Be warned, it is a research preview and a weaker model overall: it is noticeably worse at long-context reading (60k+ tokens) than this one, and the numbers on its card are honest about that. Pick this model for quality, pick Whittle-Next for speed and memory footprint, or use both.

(This comparison was written for the Whittle-Next-27B-A3B of early September 2026. That line has since moved on to Whittle-Qwen-3.8-35B-A3B; see its card for current measurements.)

Support this work

Every donation goes directly to GPU hours, and every GPU hour gets reported, including the ones that failed: ko-fi.com/davida81328

Provenance and licence

Quantised from logic65/Qwen3.8-Whittle-MoE-27B-A17.8B, which is carved from Qwen3.8-27B. Apache-2.0.