Text Generation
GGUF
llama.cpp
deepseek
deepseek-v4.1
mixture-of-experts
engram
imatrix
rtx-pro-6000
blackwell
cpu-offload
tool-use
reasoning
conversational
Instructions to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS # Run inference directly in the terminal: llama cli -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS # Run inference directly in the terminal: llama cli -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS # Run inference directly in the terminal: ./build/bin/llama cli -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
Use Docker
docker model run hf.co/prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
- Llama new
- LM Studio
- Jan
- vLLM
How to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
- Ollama
How to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with Ollama:
ollama run hf.co/prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
- Unsloth Desktop
- Pi
How to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with Docker Model Runner:
docker model run hf.co/prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
- Lemonade
How to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
Run and chat with the model
lemonade run user.DeepSeek-V4.1-Flash-1017-GGUF-IQ2_XS
List all available models
lemonade list
- Hermes Agent
How to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "prometheusAIR/DeepSeek-V4.1-Flash-1017-GGUF:IQ2_XS" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Add README.md
Browse files
README.md
CHANGED
|
@@ -22,50 +22,53 @@ quantized_by: prometheusAIR
|
|
| 22 |
|
| 23 |
# DeepSeek-V4.1-Flash-1017 GGUF: imatrix quants for one big workstation
|
| 24 |
|
| 25 |
-
|
| 26 |
built and validated on one machine: an RTX PRO 6000 96 GB, a 16 GB second card, and 128 GB of RAM. The routed
|
| 27 |
experts are split across GPU and RAM, and the ~195 GiB engram tables are memory-mapped and read from disk on demand,
|
| 28 |
so they cost disk space, not RAM.
|
| 29 |
|
| 30 |
-
|
| 31 |
-
16 retrieval and multi-hop questions exactly at depths up to 255K tokens.
|
| 32 |
|
| 33 |
> **These files need a DeepSeek-V4.1 build of llama.cpp.** Upstream llama.cpp cannot load DeepSeek-V4.1 yet
|
| 34 |
> ([#28696](https://github.com/ggml-org/llama.cpp/pull/28696) is open). See [Runtime](#runtime).
|
| 35 |
|
| 36 |
## Which one do I want?
|
| 37 |
|
| 38 |
-
| | **IQ3_XXS** (recommended) | **IQ2_XS** |
|
| 39 |
-
|---|---|---|
|
| 40 |
-
| Weights you must hold in VRAM + RAM | 193.9 GiB | 171.7 GiB |
|
| 41 |
-
| Engram tables (mmapped from disk) | 194.6 GiB | 194.6 GiB (identical files) |
|
| 42 |
-
| Total download | 388.4 GiB | 366.3 GiB |
|
| 43 |
-
| Mean KL divergence vs release experts | **0.141** | 0.183 |
|
| 44 |
-
| Median KL divergence | **0.023** | 0.033 |
|
| 45 |
-
| Same top-1 token | **88.5%** | 86.7% |
|
| 46 |
-
| Perplexity vs reference | +5.2% | +10.4% |
|
| 47 |
-
| Depth ladder (needle + multi-hop, to 255K) | **16/16** | **16/16** |
|
| 48 |
-
| Decode, reference box (see below) | 18.1-20.6 tok/s (16 at 255K) | 20.4-22.9 tok/s (17 at 255K) |
|
| 49 |
-
| Prefill, 32K-255K prompt | 475-527 tok/s | 529-592 tok/s |
|
| 50 |
|
| 51 |
- **IQ3_XXS:** pick this if your VRAM + RAM can hold ~194 GiB of weights plus context.
|
| 52 |
- **IQ2_XS:** pick this when you're ~20 GiB short. It costs 30% more KL divergence and ~1.8 points of top-1
|
| 53 |
agreement. On the same box it decodes ~10% faster and prefills ~15% faster, because fewer expert bytes sit in RAM.
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
-
Shards `-
|
| 56 |
-
|
| 57 |
-
link them under the
|
| 58 |
|
| 59 |
## Recipes
|
| 60 |
|
| 61 |
-
| tensor group | IQ3_XXS | IQ2_XS |
|
| 62 |
-
|---|---|---|
|
| 63 |
-
| Routed experts gate/up (80 tensors) | `IQ3_XXS` on 28 layers, `IQ2_XS` on 12 | `IQ2_XS` |
|
| 64 |
-
| Routed experts down (40) | `IQ3_XXS` | `IQ3_XXS` |
|
| 65 |
-
| Shared expert, attention, indexer/engram projections | `Q8_0` (copied from the source, not re-quantized) | same |
|
| 66 |
-
| Router, mHC, compressor, norms, some indexer/engram parts | `F32` / `BF16`, as in the source | same |
|
| 67 |
-
| `token_embd`, `output` | `BF16` | `BF16` |
|
| 68 |
-
| Engram tables (2 x 384M rows x 256) | `Q8_0`, see [Engram precision](#engram-precision) | same |
|
| 69 |
|
| 70 |
The routed experts are about 95% of the non-engram bytes, so their bit width decides quality.
|
| 71 |
|
|
@@ -134,6 +137,11 @@ CUDA_VISIBLE_DEVICES=0,1 llama-server \
|
|
| 134 |
(`-ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1'`). That places 77.3 GiB on GPU0 and 85.1 GiB in RAM;
|
| 135 |
GPU0 holds 90,730 MiB after load and peaks at 90,940 MiB in the 255K rung.
|
| 136 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
### Settings that matter
|
| 138 |
|
| 139 |
- **`-np 1`.** On this runtime, more than one cache stream scores tokens measurably differently from the
|
|
@@ -169,17 +177,18 @@ each (81,920 tokens), one sequence per batch.
|
|
| 169 |
**Depth ladder.** Facts planted at 10/50/90% depth in a synthetic log, and four questions per rung: a verbatim
|
| 170 |
needle, a single retrieval, a three-fact multi-hop sum, and a comparison. Thinking on, effort low. The IQ3_XXS
|
| 171 |
rows were measured before the shard-1 metadata rewrite (same tensor bytes); the release files repeated the 32K rung, 4/4.
|
|
|
|
| 172 |
|
| 173 |
-
| prompt tokens | IQ3_XXS
|
| 174 |
-
|---:|---
|
| 175 |
-
| 32,137 |
|
| 176 |
-
| 64,141 |
|
| 177 |
-
| 125,244 |
|
| 178 |
-
| 255,340 |
|
| 179 |
|
| 180 |
**Functional checks:** arithmetic, thinking open/close, tool calls with thinking on and off, and a tool-result
|
| 181 |
-
round trip all pass on
|
| 182 |
-
18.4-18.9 tok/s (IQ3_XXS)
|
| 183 |
|
| 184 |
## Calibration
|
| 185 |
|
|
@@ -199,6 +208,7 @@ sequences of 8,192 tokens. All 46,080 routed-expert slots received activations.
|
|
| 199 |
|---|---|
|
| 200 |
| IQ3_XXS | `DeepSeek-V4.1-Flash-1017-IQ3_XXS-000{01..10}-of-00010.gguf` |
|
| 201 |
| IQ2_XS | `DeepSeek-V4.1-Flash-1017-IQ2_XS-000{01..10}-of-00010.gguf` |
|
|
|
|
| 202 |
| imatrix | `imatrix/DeepSeek-V4.1-Flash-1017-imatrix.gguf` |
|
| 203 |
| runtime patch | `patches/dsv41s-runtime.patch` |
|
| 204 |
| checksums | `SHA256SUMS` (`sha256sum -c SHA256SUMS --ignore-missing`) |
|
|
|
|
| 22 |
|
| 23 |
# DeepSeek-V4.1-Flash-1017 GGUF: imatrix quants for one big workstation
|
| 24 |
|
| 25 |
+
Three imatrix-guided quants of [DeepSeek-V4.1-Flash-1017](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017),
|
| 26 |
built and validated on one machine: an RTX PRO 6000 96 GB, a 16 GB second card, and 128 GB of RAM. The routed
|
| 27 |
experts are split across GPU and RAM, and the ~195 GiB engram tables are memory-mapped and read from disk on demand,
|
| 28 |
so they cost disk space, not RAM.
|
| 29 |
|
| 30 |
+
All three quants were measured against the release-precision experts, not against another quant, and all three
|
| 31 |
+
answer all 16 retrieval and multi-hop questions exactly at depths up to 255K tokens.
|
| 32 |
|
| 33 |
> **These files need a DeepSeek-V4.1 build of llama.cpp.** Upstream llama.cpp cannot load DeepSeek-V4.1 yet
|
| 34 |
> ([#28696](https://github.com/ggml-org/llama.cpp/pull/28696) is open). See [Runtime](#runtime).
|
| 35 |
|
| 36 |
## Which one do I want?
|
| 37 |
|
| 38 |
+
| | **IQ3_XXS** (recommended) | **IQ2_XS** | **IQ2_XXS** |
|
| 39 |
+
|---|---|---|---|
|
| 40 |
+
| Weights you must hold in VRAM + RAM | 193.9 GiB | 171.7 GiB | 152.0 GiB |
|
| 41 |
+
| Engram tables (mmapped from disk) | 194.6 GiB | 194.6 GiB (identical files) | 194.6 GiB (identical files) |
|
| 42 |
+
| Total download | 388.4 GiB | 366.3 GiB | 346.5 GiB |
|
| 43 |
+
| Mean KL divergence vs release experts | **0.141** | 0.183 | 0.229 |
|
| 44 |
+
| Median KL divergence | **0.023** | 0.033 | 0.046 |
|
| 45 |
+
| Same top-1 token | **88.5%** | 86.7% | 85.1% |
|
| 46 |
+
| Perplexity vs reference | +5.2% | +10.4% | +14.1% |
|
| 47 |
+
| Depth ladder (needle + multi-hop, to 255K) | **16/16** | **16/16** | **16/16** |
|
| 48 |
+
| Decode, reference box (see below) | 18.1-20.6 tok/s (16 at 255K) | 20.4-22.9 tok/s (17 at 255K) | 22.1-25.9 tok/s (19 at 255K) |
|
| 49 |
+
| Prefill, 32K-255K prompt | 475-527 tok/s | 529-592 tok/s | 636-727 tok/s |
|
| 50 |
|
| 51 |
- **IQ3_XXS:** pick this if your VRAM + RAM can hold ~194 GiB of weights plus context.
|
| 52 |
- **IQ2_XS:** pick this when you're ~20 GiB short. It costs 30% more KL divergence and ~1.8 points of top-1
|
| 53 |
agreement. On the same box it decodes ~10% faster and prefills ~15% faster, because fewer expert bytes sit in RAM.
|
| 54 |
+
- **IQ2_XXS:** pick this when you're ~40 GiB short of IQ3_XXS, or when speed matters more than fidelity. Against
|
| 55 |
+
IQ3_XXS it costs 62% more KL divergence and 3.4 points of top-1 agreement; on the same box it decodes ~25% faster
|
| 56 |
+
and prefills ~40% faster.
|
| 57 |
|
| 58 |
+
Shards `-00008-`, `-00009-` and `-00010-` (the two engram tables among them, 204.1 GiB together) are byte-identical
|
| 59 |
+
in all three quants, and `-00006-` is also shared by IQ3_XXS and IQ2_XS (see `SHA256SUMS`). If you want more than
|
| 60 |
+
one quant, download those once and copy or link them under the other quant's name.
|
| 61 |
|
| 62 |
## Recipes
|
| 63 |
|
| 64 |
+
| tensor group | IQ3_XXS | IQ2_XS | IQ2_XXS |
|
| 65 |
+
|---|---|---|---|
|
| 66 |
+
| Routed experts gate/up (80 tensors) | `IQ3_XXS` on 28 layers, `IQ2_XS` on 12 | `IQ2_XS` | `IQ2_XXS` |
|
| 67 |
+
| Routed experts down (40) | `IQ3_XXS` | `IQ3_XXS` | `Q2_K` |
|
| 68 |
+
| Shared expert, attention, indexer/engram projections | `Q8_0` (copied from the source, not re-quantized) | same | same |
|
| 69 |
+
| Router, mHC, compressor, norms, some indexer/engram parts | `F32` / `BF16`, as in the source | same | same |
|
| 70 |
+
| `token_embd`, `output` | `BF16` | `BF16` | `BF16` |
|
| 71 |
+
| Engram tables (2 x 384M rows x 256) | `Q8_0`, see [Engram precision](#engram-precision) | same | same |
|
| 72 |
|
| 73 |
The routed experts are about 95% of the non-engram bytes, so their bit width decides quality.
|
| 74 |
|
|
|
|
| 137 |
(`-ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1'`). That places 77.3 GiB on GPU0 and 85.1 GiB in RAM;
|
| 138 |
GPU0 holds 90,730 MiB after load and peaks at 90,940 MiB in the 255K rung.
|
| 139 |
|
| 140 |
+
**IQ2_XXS on the same box:** use `-ncmoe 18` and send block 18's down-projection experts to RAM as well
|
| 141 |
+
(`-ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1,blk\.18\.ffn_down_exps\.weight=CPU'`). That places
|
| 142 |
+
78.1 GiB on GPU0, 7.1 GiB on GPU1 and 65.5 GiB in RAM; GPU0 holds 91,002 MiB after load and peaks at 91,180 MiB in
|
| 143 |
+
the 255K rung.
|
| 144 |
+
|
| 145 |
### Settings that matter
|
| 146 |
|
| 147 |
- **`-np 1`.** On this runtime, more than one cache stream scores tokens measurably differently from the
|
|
|
|
| 177 |
**Depth ladder.** Facts planted at 10/50/90% depth in a synthetic log, and four questions per rung: a verbatim
|
| 178 |
needle, a single retrieval, a three-fact multi-hop sum, and a comparison. Thinking on, effort low. The IQ3_XXS
|
| 179 |
rows were measured before the shard-1 metadata rewrite (same tensor bytes); the release files repeated the 32K rung, 4/4.
|
| 180 |
+
The IQ2_XS and IQ2_XXS rows are the release files. Every rung scored 4/4 on all three quants.
|
| 181 |
|
| 182 |
+
| prompt tokens | IQ3_XXS prefill / decode | IQ2_XS prefill / decode | IQ2_XXS prefill / decode |
|
| 183 |
+
|---:|---:|---:|---:|
|
| 184 |
+
| 32,137 | 475 / 19.2-20.1 tok/s | 592 / 21.7-22.9 tok/s | 727 / 24.2-25.1 tok/s |
|
| 185 |
+
| 64,141 | 527 / 19.2-20.6 tok/s | 591 / 20.7-22.9 tok/s | 720 / 23.9-25.9 tok/s |
|
| 186 |
+
| 125,244 | 513 / 18.1-19.5 tok/s | 570 / 20.4-21.5 tok/s | 688 / 22.1-23.8 tok/s |
|
| 187 |
+
| 255,340 | 479 / 15.6-16.1 tok/s | 529 / 16.3-17.3 tok/s | 636 / 18.8-19.4 tok/s |
|
| 188 |
|
| 189 |
**Functional checks:** arithmetic, thinking open/close, tool calls with thinking on and off, and a tool-result
|
| 190 |
+
round trip all pass on all three quants, using the release files and their embedded template. Short-prompt decode
|
| 191 |
+
is 18.4-18.9 tok/s (IQ3_XXS), 21.4-22.4 tok/s (IQ2_XS) and 24.1-24.8 tok/s (IQ2_XXS).
|
| 192 |
|
| 193 |
## Calibration
|
| 194 |
|
|
|
|
| 208 |
|---|---|
|
| 209 |
| IQ3_XXS | `DeepSeek-V4.1-Flash-1017-IQ3_XXS-000{01..10}-of-00010.gguf` |
|
| 210 |
| IQ2_XS | `DeepSeek-V4.1-Flash-1017-IQ2_XS-000{01..10}-of-00010.gguf` |
|
| 211 |
+
| IQ2_XXS | `DeepSeek-V4.1-Flash-1017-IQ2_XXS-000{01..10}-of-00010.gguf` |
|
| 212 |
| imatrix | `imatrix/DeepSeek-V4.1-Flash-1017-imatrix.gguf` |
|
| 213 |
| runtime patch | `patches/dsv41s-runtime.patch` |
|
| 214 |
| checksums | `SHA256SUMS` (`sha256sum -c SHA256SUMS --ignore-missing`) |
|