Instructions to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF # Run inference directly in the terminal: llama cli -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF # Run inference directly in the terminal: llama cli -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF # Run inference directly in the terminal: ./build/bin/llama cli -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Use Docker
docker model run hf.co/cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
- Llama new
- LM Studio
- Jan
- vLLM
How to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
- Ollama
How to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with Ollama:
ollama run hf.co/cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
- Unsloth Desktop
- Pi
How to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with Docker Model Runner:
docker model run hf.co/cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
- Lemonade
How to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Run and chat with the model
lemonade run user.DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Fixed on 2026-10-19. If you downloaded before that, re-download shards 1–7. The first upload (2026-10-18) loaded fine but generated garbage (repetition loops; wikitext-2 perplexity 16.5). The routed-expert shards
00001–00007have been requantized. Shards00008–00010,imatrix.gguf,run.shandbin/did not change, so running the samehf downloadagain fetches only the 7 new shards. Cause and fix: below.
DeepSeek-V4.1-Flash-1017 · UD-IQ2XS GGUF (experimental)
Experimental. Tested on one machine (2× RTX PRO 6000 Blackwell Max-Q) with the bundled build: wikitext-2 perplexity, a chat sanity check (arithmetic, Chinese, code, thinking on), and speed up to 14K context. No benchmark suite (MMLU, GSM8K, ...) has been run on this quant yet. Please report what you see.
GGUF quants of deepseek-ai/DeepSeek-V4.1-Flash-1017. This is a dynamic mix in the style of "UD" quants: routed experts at about 2.3–2.5 bits, everything that is small and sensitive at 6–32 bits. The aim is for all weights except the engram tables to fit on two 96 GB GPUs with room left for a long context, so that no expert has to run on the CPU.
Stock llama.cpp cannot load this model (architecture deepseek41). You need a build of
ggml-org/llama.cpp#28696 with the fixes from
cliecy/llama.cpp dsv41-1017-longctx.
A prebuilt Linux/CUDA build of that branch is included in bin/.
Quick start
hf download cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF --local-dir DeepSeek-V4.1-Flash-1017-UD-IQ2XS
bash DeepSeek-V4.1-Flash-1017-UD-IQ2XS/run.sh
# -> OpenAI-compatible API at http://127.0.0.1:8080/v1
run.sh unpacks bin/llama.cpp-dsv41-1017-linux-x64-cuda12.9.tar.zst the first time it runs (this needs zstd),
then starts llama-server. You can override these settings through environment variables:
| variable | default | meaning |
|---|---|---|
HOST, PORT |
127.0.0.1, 8080 |
listen address |
CTX |
131072 |
context length (single slot) |
API_KEY |
(none) | require this key |
GPUS |
0,1 |
CUDA_VISIBLE_DEVICES |
TS |
48,52 |
tensor split, see below |
THREADS |
physical cores, max 16 | CPU threads (the CPU only does the engram lookups) |
LOW_RAM |
0 |
1 = keep the engram tables on disk and read rows on demand (mmap). About 10 GiB RAM instead of about 115 GiB; fine on NVMe, very slow on HDD |
EXTRA_ARGS |
(none) | appended to the command line, e.g. -ctk q8_0 -ctv q8_0 |
The command run.sh runs:
llama-server -m DeepSeek-V4.1-Flash-1017-UD-IQ2XS-00001-of-00010.gguf \
-ngl 999 -fa on -sm layer -ts 48,52 -ot "engram_embd.weight=CPU" -lm none -lzm off \
-c 131072 -np 1 --jinja --chat-template-file deepseek-ai-DeepSeek-V4.1-Flash.jinja
Requirements
- GPUs: two NVIDIA GPUs with ≥ 96 GB each, e.g. 2× RTX PRO 6000 Blackwell. About 158 GiB of weights go on the GPUs, roughly 79 GiB per card. The prebuilt binaries cover sm_120 (Blackwell workstation), sm_90 (Hopper) and sm_89 (Ada).
- Driver: a driver that supports CUDA 12.9 (≥ 575 recommended; Blackwell needs ≥ 570).
- RAM: about 115 GiB free. The two engram tables (103 GiB) stay resident in RAM by default. Use
LOW_RAM=1otherwise. - Disk: 281 GB. The first load reads all of it, so it takes several minutes on NVMe and more than 30 minutes on a hard disk.
- OS: Linux x86_64 with glibc ≥ 2.28. CPU with AVX2.
Why -ts 48,52
V4.1 compresses its KV cache on a few source layers (2, 8, 14, 20), and the layers after each source read the
source's rows. With the default two-GPU split, the boundary lands at layer 21. Layer 20's compressed KV then sits
on GPU 0 while its 19 readers sit on GPU 1, and every generated token copies the whole compressed cache through
the host. That was about 340 MB per token in each direction at 16K context, and it grows with the context.
-ts 48,52 puts the boundary at layer 20 instead. On Q2_K this took generation at 16K from 23 to 34 tok/s, before
the other fixes.
Chat template
The template embedded in the GGUF is the DeepSeek-V4 one; tool calls need the V4.1 tag names. run.sh passes the
V4.1 template that ships in bin/. With thinking on, greedy decoding tends to re-check its answer until it runs
out of tokens. Use temperature 0.6 / top_p 0.95, or turn thinking off with
"chat_template_kwargs": {"enable_thinking": false}.
What is in the file
Source: the Q8_0 GGUF of mrfakename/DeepSeek-V4.1-Flash-1017-GGUF.
In that file the routed experts are native MXFP4 and the small tensors are F32/BF16. It was requantized with
llama-quantize --allow-requantize --imatrix imatrix.gguf --exclude-weights attn_output_a --tensor-type-file ...,
with the zero-sign dithering described below
(LLAMA_QUANT_DITHER_ZEROS=1) for the IQ2 expert tensors.
| tensors | type | size |
|---|---|---|
routed experts ffn_gate_exps, ffn_up_exps (40 layers × 384 experts) |
IQ2_XS | 2 × 48.8 GiB |
routed experts ffn_down_exps |
IQ2_S | 54.1 GiB |
attention attn_q_a/q_b/kv/output_a/output_b, indexer attn_q_b |
Q6_K | 3.9 GiB |
shared expert ffn_{gate,up,down}_shexp |
Q6_K | 1.1 GiB |
output |
Q6_K | 0.5 GiB |
token_embd, engram_wkv |
Q8_0 | 1.0 GiB |
router ffn_gate_inp, compressor attn_compressor_{kv,gate}, indexer proj/attn_k, engram_q/k |
BF16 | 0.2 GiB |
hyper-connection mixers hc_attn_fn, hc_ffn_fn |
F32 | 0.14 GiB |
| norms, biases, sinks, hc base/scale | F32 | < 0.01 GiB |
engram tables blk.{1,14}.engram_embd (kept on the CPU) |
Q4_K | 2 × 51.5 GiB |
| total | 261.4 GiB |
- Why
hc_*_fnis F32. These matrices produce the coefficients that mix the four residual streams around every attention and FFN sublayer, so quantization error compounds across all 40 layers. The reference keeps them in FP32, and measurements in the PR thread show BF16 already misses by 1e-2 while F32 matches to 1e-6. The published 1017 Q2_K files have them at Q2_K. - Why engram is Q4_K on the CPU. The tables are 384M rows × 256 and are read 48 rows per token, so they want host memory rather than VRAM. Q4_K keeps them accurate at 103 GiB of RAM.
- Importance matrix.
imatrix.ggufis included. It was computed with the Q2_K model on 194K tokens: a general English calibration set, Chinese Wikipedia, wikitext-2 and source code. Its PPL during collection was 6.53 (ctx 512).attn_output_a(the groupedwo_a) was quantized without the imatrix, because its 8 × 4096 entry does not match the 4096-column tensor in this llama.cpp version. - Metadata.
general.file_typereads Q8_0 because the base type passed tollama-quantizewas Q8_0. The per-tensor types above are what is actually in the file.
The llama.cpp build in bin/
bin/llama.cpp-dsv41-1017-linux-x64-cuda12.9.tar.zst contains llama-server, llama-cli, llama-bench,
llama-perplexity, llama-imatrix, llama-quantize and llama-gguf-split, plus the V4.1 chat template. It was
built from cliecy/llama.cpp dsv41-1017-longctx
(commit 1f91bd6) with CUDA 12.9 for sm_89 / sm_90 / sm_120a and an AVX2 CPU baseline. The libraries are found
through $ORIGIN, and the CUDA runtime and cuBLAS are bundled. See BUILD_INFO.txt and THIRD_PARTY_NOTICES.md
inside. The branch is PR #28696 at 2076412 plus 7 commits:
- CUDA MMQ
MUL_MAT_IDpadding fix: a backport of master #29953, against illegal-memory-access crashes on large prompt batches. - Reader layers reuse the previous index source's top-k (fix by @nicholasshirley).
- The two-level candidate mask of the reference (needed beyond 16K positions).
- The fused lightning indexer for V4.1.
- Gathering only the selected compressed rows at decode.
- Fused hyper-connection pre op kept across a GPU split (@midagedev, vcruz305/llama.cpp#2).
- An
llama-imatrixcrash fix for V4.1 graphs.
To build it yourself:
git clone -b dsv41-1017-longctx https://github.com/cliecy/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server
Measured with this file
On 2× RTX PRO 6000 Blackwell Max-Q (PCIe), with the bundled build and the run.sh settings:
| this file (fixed) | first upload (2026-10-18) | Q2_K 1017 | |
|---|---|---|---|
| wikitext-2 perplexity, 2 × 2048 | 1.62 | 16.54 | 4.69 (same build) |
| wikitext-2 perplexity, 6 × 2048 | 2.044 ± 0.045 | – | 4.749 ± 0.137 (earlier run, see below) |
- Decode: 74–75 tok/s on short chats, 71 tok/s after a 14K-token prompt. Prompt processing: 678 tok/s at 14K.
- Memory with the
run.shdefaults (CTX=131072): 81.0 GiB and 81.2 GiB VRAM on the two cards, about 105 GiB RAM for the engram tables. Loading took 30 minutes from a hard disk. LOW_RAM=1gives the same perplexity as the default mode.
The perplexity of this file is much lower than Q2_K's. The difference comes from the tensors outside the routed experts (attention, shared expert, hyper-connections, compressor, indexer), which are kept at higher precision here. The IQ2 experts themselves are slightly worse than Q2_K's: on the first 2048-token chunk this file measures 1.79, and 1.72 with the Q2_K experts swapped in.
2026-10-19 fix: why the first upload produced garbage
- The routed experts of the source are MXFP4, and about 11.6 % of their weights are exactly 0.
- IQ2_XS and IQ2_S have no zero level in their codebooks, and
llama-quantizegives a weight of exactly 0 a positive sign. So every zero became the smallest positive grid value. - Per weight, the error was normal: relative RMS error 0.31 for
gate/up, the same as Q2_K. Within each row, though, the zeros all erred the same way, so the errors added up along every row instead of cancelling. Summed over a row, the error came out at +3.6 standard deviations forgate/upand +8 fordown, positive in 100 % of rows. Independent errors would give about ±0.8, with half the rows positive. - The damage sits in the deeper layers. Swapping expert shards against the Q2_K file showed that IQ2 experts in layers 0–23 cost about 3 % perplexity, while IQ2 experts in layers 24–39 raised it from 1.72 to 6.46 (first 2048-token chunk of wikitext-2).
- Fix. Before quantizing, every exact zero gets a tiny pseudo-random sign. The tensor types, file size and
per-weight error stay the same; only the row bias is gone. The runtime and
bin/did not change. - The fix is a small patch to
llama-quant.cpp, included here asllama-quant-dither-zeros.patch. It is off unlessLLAMA_QUANT_DITHER_ZEROS=1is set. The environment variable is not part of upstream llama.cpp, and thellama-quantizeinbin/does not include the patch, so requantizing with that binary reproduces the problem. - Ruled out along the way:
- the CUDA kernels:
test-backend-opspasses, includingMUL_MAT_IDwith this model's matrix shapes and the fused MoE paths - the engram tables, and the
LOW_RAMpath - the GGUF metadata and the imatrix weighting
- the bundled binary: Q2_K runs correctly with it
- the CUDA kernels:
- Measured here for IQ2_XS and IQ2_S only. Other IQ types without a zero level (IQ2_XXS, IQ3_XXS, IQ3_S) are probably affected the same way but were not checked. When requantizing MXFP4 weights (DeepSeek V4, gpt-oss) to such types, check the per-row error sum, not only the RMS error.
Reference numbers (different quant: Q2_K, not this file)
These were measured with the published Q2_K 1017 GGUF, which needs some experts on the CPU (-ncmoe 3, or
4 CPU expert layers with -ts 48,52), on 2× RTX PRO 6000 Blackwell Max-Q over PCIe. They show what the runtime
fixes do; they are not a measurement of this UD-IQ2XS quant.
Generation speed (tok/s) by context depth:
| context | stock branch, default split | -ts 48,52 |
-ts 48,52 + all fixes |
|---|---|---|---|
| 512 | 44.9 | 40.8 | 42.2 |
| 4K | 36.5 | 39.8 | 47.3 |
| 16K | 23.1 | 34.2 | 45.3 |
| 32K | – | 28.6 | 47.0 |
| 64K | – | 21.7 | 27.9 |
| 120K | – | – | 18.8 |
- Prompt processing: 430 tok/s at 4K, 275 at 64K, 200 at 120K.
- wikitext-2 perplexity (6 × 2048): 4.749 ± 0.137.
- Needle in a haystack at 48.5K tokens, needle at 25 % and at 60 %: both correct.
- The drop at ≥ 64K comes from
top_kover the compressed positions. With CUDA 12.x it falls back to a full sort; a CUDA 13 build (CCCL ≥ 3.2) should avoid that but has not been measured.
Credits
- deepseek-ai for the model.
- @mrfakename for the 1017 GGUF conversion used as the source.
- @vcruz305 for the V4.1 conversion and runtime in ggml-org/llama.cpp#28696.
- @midagedev for the DSpark/HC-pre fixes and the V4.1 chat template.
- @nicholasshirley for the reused top-k fix.
- Mermiges for the MMQ out-of-bounds analysis.
Quantized and packaged with the help of an AI coding assistant (Claude).
The llama.cpp fixes in bin/
The bundled llama.cpp is DeepSeek V4.1 support from ggml-org/llama.cpp#28696 (not yet merged) plus 7 patches for long context and multi-GPU (two-level candidate mask, fused lightning indexer, decode-time gather of the selected compressed KV rows, top-k reuse on reader layers, fused HC pre across a GPU split, an imatrix guard, an MMQ padding backport). On a 2-GPU Q2_K run they raise single-stream decode at 16K context from 23 to 45 tok/s. Patch files, build steps and measurements: LLAMACPP-FIXES.md (中文); source branch: cliecy/llama.cpp@dsv41-1017-longctx.
- Downloads last month
- 819
We're not able to determine the quantization variants.
Model tree for cliecy/DeepSeek-V4.1-Flash-1017-UD-IQ2XS-GGUF
Base model
deepseek-ai/DeepSeek-V4.1-Flash-1017