Qwen3.8-27B MLX Hybrid Q8/Q4 + Vision

adrianmurray/Qwen3.8-27B-Hybrid-Q8Q4 with the vision tower grafted back on. The language model weights are Adrian Murray's, bit-for-bit — this repo adds the 333 model.visual.* tensors from the official Qwen/Qwen3.8-27B checkpoint and nothing else.

Why this exists

mlx_lm.convert drops every vision tensor unconditionally (mlx_lm/models/qwen3_5.py:387), so any Qwen3.8-27B converted with it is text-only. The language model is intact; the ViT is simply absent, along with vision_config. This repo restores it.

One directory now serves both paths:

loader uses result
mlx_lm / qwen-prime-runtime drops vision_tower.* at load text, full speed, speculative decoding intact
mlx-vlm (>= 0.6.15) loads vision_tower.* image + text

Measured (Apple M1 Max, 64 GB)

workload speed
text, DFlash speculative decoding 22.0–22.3 tok/s (58.8% acceptance)
image, mlx-vlm 17.6–18.1 tok/s decode, 46.9–95.7 tok/s prefill
peak memory with an image 22.3 GB

Text throughput is unchanged from Adrian's original — the extra tensors are ignored by the text path, so the graft is free.

Requirements

mlx-vlm >= 0.6.15. Earlier versions (including 0.6.3) apply a +1.0 norm shift unconditionally in the qwen3_5 sanitizer, which double-shifts weights that are already in post-sanitize form and silently corrupts the model — it still loads and still answers. 0.6.15 added the should_shift_norm_weights guard that mlx_lm 0.31.3 has.

Built and verified with mlx==0.32.0, mlx-lm==0.31.3, mlx-vlm==0.6.15.

How it was built

  1. All 333 model.visual.* tensors live in shard 1 of 18 of the base checkpoint (460.7 M params, 0.92 GB bf16), so only ~4 GB had to be fetched, not 54 GB.
  2. Renamed to post-sanitize form (model.visual.* → vision_tower.*) to match how Adrian's language weights are already stored (language_model.model.*). mlx_vlm's sanitize_key passes vision_tower. through untouched.
  3. Language-model shards copied byte-for-byte from Adrian's repo.
  4. config.json = Adrian's, plus the base vision_config, plus language_model_only: false. His per-module quantization dict is carried over verbatim.

The vision tower is deliberately not quantized — it is 460 M params and runs once per image at prefill rather than per generated token, so quantizing it would save ~0.4 GB and no decode time while risking ViT quality. mlx_vlm's class predicate falls back to f"{p}.scales" in weights, so unlisted modules are correctly left alone.

Verification

The graft is checked two ways, because mlx_vlm silently skips mis-keyed vision tensors rather than erroring — a randomly-initialized tower still produces fluent, plausible answers about an image:

  1. Unguessable OCR. A synthetic invoice with a random code; the model returned QK7-95KW4M, 14 March 2027, Bay 12, Kestrel Wharf, 1,483.62 — all exact.
  2. Tensor audit. 333/333 vision_tower.* tensors live in the loaded model, none all-zero, spot-checked byte-identical to the shard on disk.

A "describe the shapes" test is not sufficient — shape, colour and position are guessable from the prompt alone.

Credit

The quantization — the hard part, and the reason this runs at 22 tok/s — is entirely Adrian Murray's. See adrianmurray/Qwen3.8-27B-Hybrid-Q8Q4 for the scheme, and adrianmurray/Qwen3.8-27B-MTP-MLX-6bit for the matching native-MTP draft model.

Original model: Qwen/Qwen3.8-27B © Alibaba Cloud, Apache 2.0.

Downloads last month
39
Safetensors
Model size
27B params
Tensor type
F16
·
BF16
·
U32
·
MLX
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision