Instructions to use XcaliburMoon/muse-glimmer-30b-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use XcaliburMoon/muse-glimmer-30b-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16 # Run inference directly in the terminal: llama cli -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16 # Run inference directly in the terminal: llama cli -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16
Use Docker
docker model run hf.co/XcaliburMoon/muse-glimmer-30b-gguf:BF16
- Llama new
- LM Studio
- Jan
- Ollama
How to use XcaliburMoon/muse-glimmer-30b-gguf with Ollama:
ollama run hf.co/XcaliburMoon/muse-glimmer-30b-gguf:BF16
- Unsloth Desktop
- Pi
How to use XcaliburMoon/muse-glimmer-30b-gguf with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "XcaliburMoon/muse-glimmer-30b-gguf:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use XcaliburMoon/muse-glimmer-30b-gguf with Docker Model Runner:
docker model run hf.co/XcaliburMoon/muse-glimmer-30b-gguf:BF16
- Lemonade
How to use XcaliburMoon/muse-glimmer-30b-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull XcaliburMoon/muse-glimmer-30b-gguf:BF16
Run and chat with the model
lemonade run user.muse-glimmer-30b-gguf-BF16
List all available models
lemonade list
- Hermes Agent
How to use XcaliburMoon/muse-glimmer-30b-gguf with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default XcaliburMoon/muse-glimmer-30b-gguf:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use XcaliburMoon/muse-glimmer-30b-gguf with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf XcaliburMoon/muse-glimmer-30b-gguf:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "XcaliburMoon/muse-glimmer-30b-gguf:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Muse-Glimmer-30B GGUFs (Q4_K_M + vision + drafter)
Runnable llama.cpp weights for Muse-Glimmer-30B, quantised to Q4_K_M and published together with the small side files that the model needs in order to actually load: the vision projector, the speculative drafter, the chat template and the tokenizer. Everything here is Apache-2.0, same as the base model.
The point of this repo is that a fresh machine needs one hf download and
nothing else. There is no separate tokenizer fetch, no template to copy out of
another repo, no "which mmproj goes with this" guesswork.
What is in here
| File | Bytes | Size | What it is |
|---|---|---|---|
muse-Q4_K_M.gguf |
16,935,291,424 | 15.8 GiB | The model. Quantised here from the BF16 GGUF with llama-quantize. |
mmproj-Muse-Glimmer-30B-BF16.gguf |
3,849,173,728 | 3.6 GiB | Vision projector, BF16. Required for image input. |
dflash-kquant.gguf |
1,631,205,312 | 1.5 GiB | DFlash speculative drafter. Optional, for draft decoding. |
chat_template.jinja |
small | Canonical chat template. Needed by --jinja. |
|
tokenizer.json |
small | Tokenizer. | |
tokenizer_config.json |
small | Tokenizer config. | |
config.json |
small | Model config as published by the base repo. | |
SHA256SUMS |
small | Checksums for the three large files. | |
run.sh |
small | Launcher: fetches what it needs, execs llama-server with the right flags. |
|
LICENSE |
small | Apache-2.0, inherited from the base model. |
Total download for a working text model: one file, 15.8 GiB. Add the mmproj for vision, add the drafter for speculative decoding.
The BF16 GGUF is not mirrored here. It is 51.9 GiB across two shards and the authoritative copy already exists upstream; mirroring it would only risk drift.
Quick start
The short path. run.sh fetches what it needs and execs llama-server:
hf download XcaliburMoon/muse-glimmer-30b-gguf run.sh --local-dir muse
chmod +x muse/run.sh && ./muse/run.sh
Knobs are environment variables, all optional:
VISION=1 ./muse/run.sh # add the vision projector
DRAFT=1 ./muse/run.sh # add the speculative drafter
SLOTS=16 CTX=131072 PORT=8000 ./muse/run.sh
If you would rather drive llama.cpp yourself, download the files and use these. Text only, 8k context:
llama-server \
-m muse/muse-Q4_K_M.gguf \
-c 8192 -ngl 99 -fa on --jinja \
--chat-template-file muse/chat_template.jinja \
--alias muse-glimmer-30b-q4 \
--host 127.0.0.1 --port 8080
This is the configuration that was actually measured below. For a multi-user
endpoint, raise the total context and split it over slots. llama.cpp divides
-c by --parallel, so -c 131072 --parallel 16 gives 16 slots of 8192 each:
llama-server \
-m muse/muse-Q4_K_M.gguf \
-c 131072 --parallel 16 -ngl 99 -fa on --jinja \
--chat-template-file muse/chat_template.jinja \
--alias muse-glimmer-30b-q4 --metrics \
--host 0.0.0.0 --port 8080
Add vision:
--mmproj muse/mmproj-Muse-Glimmer-30B-BF16.gguf
Add speculative decoding:
-md muse/dflash-kquant.gguf --draft-max 8 --draft-min 2
--metrics exposes /metrics and /health. While the weights load,
/health answers 503 and then flips to 200; gate any automation on that rather
than on a sleep.
Measured numbers
One A100 SXM4 80GB, llama.cpp build 3c809f4 (ggml 0.28.0), CUDA, full offload,
flash attention on, 2026-10-29. These are the server's own reported timings, not
estimated.
| Q4_K_M | BF16 | |
|---|---|---|
| File size | 15.8 GiB | 51.9 GiB |
| VRAM resident | 19.3 GB | does not fit with room to spare |
| Decode, single stream | 52.8 tok/s | 28.5 tok/s |
| Prefill, single stream | 1306 tok/s | - |
| Decode at 16 concurrent | 351 tok/s aggregate | - |
Q4_K_M is 1.92x the BF16 decode rate on this card, and prefill stays flat out to 16 concurrent slots, which is what makes a 16-slot deployment reasonable on a single 80GB device.
A call with a cold 627-token prompt measured 1172 tok/s prefill, 627 of 627 tokens uncached, 535 ms to first token, 52.4 tok/s decode. A repeated prompt with a shared prefix served 35 of 51 prompt tokens from cache, so prefix reuse is worth keeping warm if you run the same system prompt constantly.
Caveat worth stating: 52.8 tok/s is the card's decode rate. If the client reaches the server through an SSH tunnel or a proxy, first-token latency picks up 0.7 to 2.8 s of hop time. Decode is unaffected. Publish a port or run the client on the host if that matters.
Chat template and reasoning
The model thinks before it answers. Use --jinja with the shipped
chat_template.jinja or the reasoning channel will be mangled and you will get
empty replies.
Two llama.cpp behaviours bite on a thinking model:
- llama.cpp counts reasoning tokens against
max_tokens. Budgetanswer_tokens + think_tokens, or a smallmax_tokensreturns an empty answer because the model spent the whole budget thinking. - An empty
contentwith reasoning present is a truncation, not a model failure. Readreasoning_contentbefore concluding the server is broken.
Reasoning effort is a request-level field rather than a separate model:
{"model": "muse-glimmer-30b-q4", "messages": [...], "reasoning_strength": "high"}
Tool calling is native. Pass tools in the OpenAI schema; the template emits the
calls in the model's own format.
Provenance
- Base model:
meta-models/Muse-Glimmer-30B, Apache-2.0, ungated. - BF16 GGUF and the mmproj and dflash side files originate from the upstream GGUF release of that model.
muse-Q4_K_M.ggufwas quantised here withllama-quantizefrom that BF16 GGUF atQ4_K_M, then verified by serving it and reading/healthplus a full generation.- Checksums in
SHA256SUMSwere taken from the exact bytes that were uploaded, on the host that uploaded them.
If you need a different quantisation, quantise from the upstream BF16 rather than requantising the Q4 file here.
Licence
Apache-2.0, inherited from the base model. LICENSE is included. No additional
restriction is placed on these files beyond that licence.
- Downloads last month
- 666
4-bit
Model tree for XcaliburMoon/muse-glimmer-30b-gguf
Base model
meta-models/Muse-Glimmer-30B