Instructions to use ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision") config = load_config("ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "ahpramesi/Qwen3.8-27B-Hybrid-Q8Q4-Vision" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Qwen3.8-27B MLX Hybrid Q8/Q4 + Vision
adrianmurray/Qwen3.8-27B-Hybrid-Q8Q4
with the vision tower grafted back on. The language model weights are Adrian
Murray's, bit-for-bit — this repo adds the 333 model.visual.* tensors from the
official Qwen/Qwen3.8-27B checkpoint and nothing else.
Why this exists
mlx_lm.convert drops every vision tensor unconditionally
(mlx_lm/models/qwen3_5.py:387), so any
Qwen3.8-27B converted with it is text-only. The language model is intact; the ViT
is simply absent, along with vision_config. This repo restores it.
One directory now serves both paths:
| loader | uses | result |
|---|---|---|
mlx_lm / qwen-prime-runtime |
drops vision_tower.* at load |
text, full speed, speculative decoding intact |
mlx-vlm (>= 0.6.15) |
loads vision_tower.* |
image + text |
Measured (Apple M1 Max, 64 GB)
| workload | speed |
|---|---|
| text, DFlash speculative decoding | 22.0–22.3 tok/s (58.8% acceptance) |
| image, mlx-vlm | 17.6–18.1 tok/s decode, 46.9–95.7 tok/s prefill |
| peak memory with an image | 22.3 GB |
Text throughput is unchanged from Adrian's original — the extra tensors are ignored by the text path, so the graft is free.
Requirements
mlx-vlm >= 0.6.15. Earlier versions (including 0.6.3) apply a +1.0 norm
shift unconditionally in the qwen3_5 sanitizer, which double-shifts weights
that are already in post-sanitize form and silently corrupts the model — it still
loads and still answers. 0.6.15 added the should_shift_norm_weights guard that
mlx_lm 0.31.3 has.
Built and verified with mlx==0.32.0, mlx-lm==0.31.3, mlx-vlm==0.6.15.
How it was built
- All 333
model.visual.*tensors live in shard 1 of 18 of the base checkpoint (460.7 M params, 0.92 GB bf16), so only ~4 GB had to be fetched, not 54 GB. - Renamed to post-sanitize form (
model.visual.*→vision_tower.*) to match how Adrian's language weights are already stored (language_model.model.*).mlx_vlm'ssanitize_keypassesvision_tower.through untouched. - Language-model shards copied byte-for-byte from Adrian's repo.
config.json= Adrian's, plus the basevision_config, pluslanguage_model_only: false. His per-modulequantizationdict is carried over verbatim.
The vision tower is deliberately not quantized — it is 460 M params and runs
once per image at prefill rather than per generated token, so quantizing it would
save ~0.4 GB and no decode time while risking ViT quality. mlx_vlm's class
predicate falls back to f"{p}.scales" in weights, so unlisted modules are
correctly left alone.
Verification
The graft is checked two ways, because mlx_vlm silently skips mis-keyed
vision tensors rather than erroring — a randomly-initialized tower still produces
fluent, plausible answers about an image:
- Unguessable OCR. A synthetic invoice with a random code; the model returned
QK7-95KW4M,14 March 2027,Bay 12, Kestrel Wharf,1,483.62— all exact. - Tensor audit. 333/333
vision_tower.*tensors live in the loaded model, none all-zero, spot-checked byte-identical to the shard on disk.
A "describe the shapes" test is not sufficient — shape, colour and position are guessable from the prompt alone.
Credit
The quantization — the hard part, and the reason this runs at 22 tok/s — is
entirely Adrian Murray's. See
adrianmurray/Qwen3.8-27B-Hybrid-Q8Q4
for the scheme, and
adrianmurray/Qwen3.8-27B-MTP-MLX-6bit
for the matching native-MTP draft model.
Original model: Qwen/Qwen3.8-27B © Alibaba Cloud, Apache 2.0.
- Downloads last month
- 39
4-bit