Muse-Glimmer-30B GGUFs (Q4_K_M + vision + drafter)

Runnable llama.cpp weights for Muse-Glimmer-30B, quantised to Q4_K_M and published together with the small side files that the model needs in order to actually load: the vision projector, the speculative drafter, the chat template and the tokenizer. Everything here is Apache-2.0, same as the base model.

The point of this repo is that a fresh machine needs one hf download and nothing else. There is no separate tokenizer fetch, no template to copy out of another repo, no "which mmproj goes with this" guesswork.

What is in here

File Bytes Size What it is
muse-Q4_K_M.gguf 16,935,291,424 15.8 GiB The model. Quantised here from the BF16 GGUF with llama-quantize.
mmproj-Muse-Glimmer-30B-BF16.gguf 3,849,173,728 3.6 GiB Vision projector, BF16. Required for image input.
dflash-kquant.gguf 1,631,205,312 1.5 GiB DFlash speculative drafter. Optional, for draft decoding.
chat_template.jinja small Canonical chat template. Needed by --jinja.
tokenizer.json small Tokenizer.
tokenizer_config.json small Tokenizer config.
config.json small Model config as published by the base repo.
SHA256SUMS small Checksums for the three large files.
run.sh small Launcher: fetches what it needs, execs llama-server with the right flags.
LICENSE small Apache-2.0, inherited from the base model.

Total download for a working text model: one file, 15.8 GiB. Add the mmproj for vision, add the drafter for speculative decoding.

The BF16 GGUF is not mirrored here. It is 51.9 GiB across two shards and the authoritative copy already exists upstream; mirroring it would only risk drift.

Quick start

The short path. run.sh fetches what it needs and execs llama-server:

hf download XcaliburMoon/muse-glimmer-30b-gguf run.sh --local-dir muse
chmod +x muse/run.sh && ./muse/run.sh

Knobs are environment variables, all optional:

VISION=1 ./muse/run.sh              # add the vision projector
DRAFT=1  ./muse/run.sh              # add the speculative drafter
SLOTS=16 CTX=131072 PORT=8000 ./muse/run.sh

If you would rather drive llama.cpp yourself, download the files and use these. Text only, 8k context:

llama-server \
  -m muse/muse-Q4_K_M.gguf \
  -c 8192 -ngl 99 -fa on --jinja \
  --chat-template-file muse/chat_template.jinja \
  --alias muse-glimmer-30b-q4 \
  --host 127.0.0.1 --port 8080

This is the configuration that was actually measured below. For a multi-user endpoint, raise the total context and split it over slots. llama.cpp divides -c by --parallel, so -c 131072 --parallel 16 gives 16 slots of 8192 each:

llama-server \
  -m muse/muse-Q4_K_M.gguf \
  -c 131072 --parallel 16 -ngl 99 -fa on --jinja \
  --chat-template-file muse/chat_template.jinja \
  --alias muse-glimmer-30b-q4 --metrics \
  --host 0.0.0.0 --port 8080

Add vision:

  --mmproj muse/mmproj-Muse-Glimmer-30B-BF16.gguf

Add speculative decoding:

  -md muse/dflash-kquant.gguf --draft-max 8 --draft-min 2

--metrics exposes /metrics and /health. While the weights load, /health answers 503 and then flips to 200; gate any automation on that rather than on a sleep.

Measured numbers

One A100 SXM4 80GB, llama.cpp build 3c809f4 (ggml 0.28.0), CUDA, full offload, flash attention on, 2026-10-29. These are the server's own reported timings, not estimated.

Q4_K_M BF16
File size 15.8 GiB 51.9 GiB
VRAM resident 19.3 GB does not fit with room to spare
Decode, single stream 52.8 tok/s 28.5 tok/s
Prefill, single stream 1306 tok/s -
Decode at 16 concurrent 351 tok/s aggregate -

Q4_K_M is 1.92x the BF16 decode rate on this card, and prefill stays flat out to 16 concurrent slots, which is what makes a 16-slot deployment reasonable on a single 80GB device.

A call with a cold 627-token prompt measured 1172 tok/s prefill, 627 of 627 tokens uncached, 535 ms to first token, 52.4 tok/s decode. A repeated prompt with a shared prefix served 35 of 51 prompt tokens from cache, so prefix reuse is worth keeping warm if you run the same system prompt constantly.

Caveat worth stating: 52.8 tok/s is the card's decode rate. If the client reaches the server through an SSH tunnel or a proxy, first-token latency picks up 0.7 to 2.8 s of hop time. Decode is unaffected. Publish a port or run the client on the host if that matters.

Chat template and reasoning

The model thinks before it answers. Use --jinja with the shipped chat_template.jinja or the reasoning channel will be mangled and you will get empty replies.

Two llama.cpp behaviours bite on a thinking model:

  • llama.cpp counts reasoning tokens against max_tokens. Budget answer_tokens + think_tokens, or a small max_tokens returns an empty answer because the model spent the whole budget thinking.
  • An empty content with reasoning present is a truncation, not a model failure. Read reasoning_content before concluding the server is broken.

Reasoning effort is a request-level field rather than a separate model:

{"model": "muse-glimmer-30b-q4", "messages": [...], "reasoning_strength": "high"}

Tool calling is native. Pass tools in the OpenAI schema; the template emits the calls in the model's own format.

Provenance

  • Base model: meta-models/Muse-Glimmer-30B, Apache-2.0, ungated.
  • BF16 GGUF and the mmproj and dflash side files originate from the upstream GGUF release of that model.
  • muse-Q4_K_M.gguf was quantised here with llama-quantize from that BF16 GGUF at Q4_K_M, then verified by serving it and reading /health plus a full generation.
  • Checksums in SHA256SUMS were taken from the exact bytes that were uploaded, on the host that uploaded them.

If you need a different quantisation, quantise from the upstream BF16 rather than requantising the Q4 file here.

Licence

Apache-2.0, inherited from the base model. LICENSE is included. No additional restriction is placed on these files beyond that licence.

Downloads last month
666
GGUF
Model size
28B params
Architecture
muse-glimmer
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for XcaliburMoon/muse-glimmer-30b-gguf

Quantized
(203)
this model