You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This model has its safety alignment removed. Read the source model's disclaimer before you use it. Access is approved automatically.
Log in or Sign Up to review the conditions and access this model content.
DeepSeek-V4.1-Flash-1017 Abliterated, pruned KT quant for 128 GB machines
A GGUF of orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF for ik_llama.cpp, sized for 128 GB of unified memory or RAM plus VRAM. The weights that stay in memory are 105.0 GiB. The Engram table, 196.6 billion weights of n-gram memory, stays on disk and is read as needed, and you can pick its precision (see below).
The routed experts are pruned from 384 to 326 per layer and quantized to IQ1_KT and IQ2_KT. Against Orcarouter's Mix-256G build, its mean KL divergence is 0.367, against 0.516 for Orcarouter's Mix-128G, which is the larger of the two in memory (see Quality).
The source model has its safety alignment removed, and quantization does not restore it. Read the source card's disclaimer before use.
It needs ik_llama.cpp. Mainline llama.cpp can't load it. The routed experts use ik_llama.cpp's KT types, and the Engram tensors use ik_llama.cpp's layout. Tested at e34d163.
Hugging Face's download widget does not recognize _KT GGUFs. Download links are in the table below.
Choosing an Engram table
The Engram table is split 2 of 7, and it comes in three precisions. Every other split is shared. Download the six shared splits and one Engram split into the same directory.
The table is read from disk a few rows per token (-dple), so its size costs disk space, not memory. In one run per table, the three used about the same memory and ran at about the same speed.
| Engram table | Download, total | Perplexity | KLD against the Q8_0 table | Same top token |
|---|---|---|---|---|
| Q8_0 | 299.6 GiB | 3.0680 | reference | reference |
| Q6_K | 255.2 GiB | 3.0769 | 0.0116 | 96.28% |
| IQ4_NL | 208.0 GiB | 3.1132 | 0.0259 | 94.50% |
Perplexity uses Orcarouter's method, on characters 0 to 300,000 of the wikitext-2 test set in 32 chunks of 2048 tokens. The KL divergence uses the same text, against the logits of this model with the Q8_0 table, so it measures the Engram table alone.
The Q8_0 and Q6_K tables were quantized from the FP8 Engram table in deepseek-ai/DeepSeek-V4.1-Flash-1017. The IQ4_NL table is the one in Orcarouter's GGUF. It is the official table quantized to IQ4_NL by llama.cpp, checked on 3,072 rows spread over both Engram layers. The abliteration did not change the Engram table in any row checked.
Files
Access is gated, and approval is automatic. Accept the terms on this page and log in with hf auth login before you download.
With the hf CLI, for the Q8_0 table:
hf download ji-farthing/DeepSeek-V4.1-Flash-1017-Abliterated-ik-llama-GGUF \
--include "DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-0000[134567]-of-00007.gguf" \
--include "engram-Q8_0/*" \
--local-dir DeepSeek-V4.1-B326u
mv DeepSeek-V4.1-B326u/engram-Q8_0/*.gguf DeepSeek-V4.1-B326u/
Use engram-Q6_K/* or engram-IQ4_NL/* for the other tables. ik_llama.cpp finds the splits by name, so the Engram split must sit next to the others.
Quality
Against Orcarouter's Mix-256G (converted to ik_llama.cpp's layout), on the same wikitext-2 text and chunks as above. All three files in this table use the IQ4_NL Engram table.
| Model | Weights in memory | Mean KLD | Median KLD | Same top token | Perplexity |
|---|---|---|---|---|---|
| Mix-256G (reference) | 2.3602 | ||||
| Mix-128G | 105.8 GiB | 0.516 | 0.129 | 78.18% | 3.551 |
| This model | 105.0 GiB | 0.367 | 0.074 | 81.86% | 3.113 |
With the Q8_0 Engram table, the perplexity of this model is 3.068.
Tool calls work with the embedded chat template. In nine agentic coding runs with tools and thinking on, all 334 tool calls came back as structured calls, and none appeared as text in the reply.
Pruning keeps the experts that the calibration text (see below) routes to most. Text far from that mix, such as languages other than English and Chinese, may lose more than these measurements show.
Running
llama-server -m DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00001-of-00007.gguf \
-c 262144 -fa on -ngl 999 -np 1 -b 2048 -ub 2048 --cache-ram 0 \
--swa-compress --ctx-checkpoints 8 -ctk q8_0 -ictk q8_0 -fidx -dple \
--jinja -ptcall --top-k 2000
This is the command the measurements on this card used. -dple keeps the Engram table on disk instead of in memory (Linux and Windows). --cache-ram 0 turns off the prompt cache in RAM, which otherwise takes up to 8 GiB. -ptcall allows parallel tool calls. --top-k 2000 leaves the sampling to temperature and top-p.
The GGUF sets DeepSeek's recommended sampling, temperature 1.0 and top-p 1.0, as the defaults.
How it was made
Source: Orcarouter's Mix-256G build of the abliterated model, converted to ik_llama.cpp's tensor layout. The conversion copies every tensor identically. It splits the Engram table into one tensor per layer and maps the Engram gate to ik_llama.cpp's form.
An importance matrix and per-expert routing counts came from one pass over 24 chunks of 8192 tokens. The text is a sample built as 45% agentic technical English, 30% code, 15% English, and 10% Chinese, and the pass used its first 96%. In each layer, the 326 experts with the highest routing counts were kept.
| Tensors | Type |
|---|---|
Routed experts (ffn_gate_exps, ffn_up_exps) |
IQ1_KT; IQ2_KT in layers 24 to 31 |
Routed experts (ffn_down_exps) |
IQ1_KT; IQ2_KT in layers 23 to 35 |
attn_q_b, attn_output_b, shared experts (ffn_*_shexp) |
IQ4_KT |
| Engram table | Q8_0, Q6_K, or IQ4_NL (see above) |
| All other tensors | as in Mix-256G: Q8_0, Q6_K, Q5_K, BF16, F32 |
The routed experts were quantized with llama-quantize --cuda-quantize from ik_llama.cpp at 804ee29, with the importance matrix.
Chat template
This GGUF embeds the DeepSeek V4.1 chat template of ik_llama.cpp at 39ccfcb (PR #2758). The source GGUF embeds a template that ik_llama.cpp's template engine does not load. ik_llama.cpp builds before 39ccfcb ship an older version of this template, and the server uses the one in the GGUF, so you don't need --chat-template-file.
Through ik_llama.cpp's server, the template matched the output of DeepSeek's reference encoder (encoding/encoding.py in deepseek-ai/DeepSeek-V4.1-Flash-1017) in 28 of 28 test conversations, including DeepSeek's own examples, reasoning effort, tool definitions, and parallel and out-of-order tool results. The three cases not run were a task token, image input, and tools declared in a second system message. Run the server with --jinja.
SHA-256
SHA-256 of the files as uploaded:
a52bc64cd3bf36240e5bad160d39ac475e92e94f24ee0106ce6c983a11d2b133 DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00001-of-00007.gguf
56b34403dceaf1300d048047699725306aa29bdb6f5728e44ac3a9314852f47a DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00003-of-00007.gguf
af3cab457b73d15116db0e35eebd77eeb83c8339b5e14fcab926c29c888c64a2 DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00004-of-00007.gguf
45df91fc8924047743953ae79a2936920fa707eea788334d0919f899f170c146 DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00005-of-00007.gguf
99188941a2d18abad9db8ecb5f7b82e28d860872a2ac66bf6e0f6dfb7815c351 DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00006-of-00007.gguf
c29f9664337c788d114271560bc40b2e2c82ad763da8bdbc8c6c236c5c423899 DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00007-of-00007.gguf
b01aee3a8a2e33409077d169288c3c1cccc92b6be5eb4671e02650c55820cdfe engram-IQ4_NL/DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00002-of-00007.gguf
6a1dcbb30dfda49ee08423b7e37e5b0f83449dcf382969f5d51fa3df309385ed engram-Q6_K/DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00002-of-00007.gguf
e14e47ddd51321376a94c9a57b87f26662aabd303ff9b8817a89ea5fd8f8d055 engram-Q8_0/DeepSeek-V4.1-Flash-1017-abliterated-B326u-KT-00002-of-00007.gguf
License
MIT, inherited from deepseek-ai/DeepSeek-V4.1-Flash-1017. The abliteration is Orcarouter's work. Their card describes how it was done and the conditions they released it under.
- Downloads last month
- 65
Model tree for ji-farthing/DeepSeek-V4.1-Flash-1017-Abliterated-ik-llama-GGUF
Base model
deepseek-ai/DeepSeek-V4.1-Flash-1017