deucebucket/Qwen3.5-122B-A10B-Cerebellum-GGUF

Qwen3.5-122B-A10B-Cerebellum

Sensitivity-guided mixed-precision quantization of Qwen/Qwen3.5-122B-A10B. 122 billion parameters, 10 billion active per token. Hybrid SSM/attention MoE architecture with 256 experts per layer, 8 active.

Compressed from 228 GB (BF16) to 32.7 GB at 2.30 bits per weight. Runs on a single RTX 3090 (24 GB VRAM + 34 GB system RAM) at 25 tokens/second.

Quantization Strategy

Single-pass from BF16 safetensors using llama.cpp llama-quantize:

• Base: IQ2_XXS across all tensors
• Promotion: ffn_down_exps elevated to Q2_K (these carry the actual expert computation)
• No demotions — IQ1_S on expert weights destroys MoE routing quality

This targets the projection weights that matter most for output quality while keeping the model small enough for consumer hardware.

Benchmarks

Measured directly on this GGUF with llama.cpp on RTX 3090. The model-index metadata in this card's frontmatter mirrors these numbers.

Benchmark	Score
Perplexity (wikitext-2, 2048 ctx)	6.73
HellaSwag	92.04%
ARC-Challenge	95.56%
EvalPlus HumanEval (base)	59.8%
EvalPlus HumanEval+	54.9%

Unquantized Qwen 3.5 122B-A10B baseline for comparison: HellaSwag ~95%, ARC ~97% (estimated from Qwen's published results on similar benchmarks). Reasoning capability is well-preserved; code generation takes the expected hit at 2-bit precision.

Recommended Settings

Temperature: 0.6 for general use, 0.0 for benchmarks and deterministic tasks.

Thinking mode: This model supports Qwen's thinking/reasoning mode. For general chat, leave it enabled. For structured tasks (benchmarks, classification, code completion), disable it:

--chat-template-kwargs '{"enable_thinking":false}' --reasoning off

Running on RTX 3090 (25 tok/s)

llama-server \
  --model Qwen3.5-122B-A10B-Cerebellum.gguf \
  --port 8084 \
  --n-gpu-layers 48 \
  --override-tensor ".*ffn_gate_exps.*=CPU,.*ffn_up_exps.*=CPU" \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --flash-attn on \
  --parallel 4

This keeps KV cache and attention layers on GPU while offloading the large expert gate/up matrices to system RAM. The 10B active parameters per token means only a fraction of the 122B total moves through the GPU each forward pass.

Memory usage: ~18 GB VRAM, ~34 GB system RAM.

Hardware requirements:

• Minimum: 24 GB VRAM GPU + 40 GB RAM
• Recommended: RTX 3090/4090 + 64 GB RAM for comfortable headroom

Build History

This is the result of iterative experimentation on this architecture:

• v1 — Uniform IQ2_XXS baseline (34 GB). Functional but no targeted optimization.
• v3 — Attempted IQ1_S demotions on gate/up expert weights. PPL exploded to 8.02. Proved that MoE expert weights cannot survive sub-2-bit quantization.
• v4 (this release) — Promoted ffn_down_exps to Q2_K, no demotions. 32.7 GB, strong reasoning scores, best balance of size and quality.

Key insight: MoE models are fragile in the opposite way from dense models. In dense architectures, FFN gate weights can be crushed freely. In MoE, the expert weights carry everything — crush those and routing falls apart.

About Cerebellum

Cerebellum is a sensitivity-guided quantization methodology. Instead of uniform compression, it measures which weight groups survive extreme quantization and which need protection, then allocates precision accordingly. The goal is making large models accessible on consumer hardware without destroying the capabilities that make them worth running.

More models: huggingface.co/deucebucket

License

Same as the base model: Apache 2.0