prometheusAIR commited on
Commit
30cab4a
·
verified ·
1 Parent(s): 493e331

Add README.md

Browse files
Files changed (1) hide show
  1. README.md +44 -34
README.md CHANGED
@@ -22,50 +22,53 @@ quantized_by: prometheusAIR
22
 
23
  # DeepSeek-V4.1-Flash-1017 GGUF: imatrix quants for one big workstation
24
 
25
- Two imatrix-guided quants of [DeepSeek-V4.1-Flash-1017](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017),
26
  built and validated on one machine: an RTX PRO 6000 96 GB, a 16 GB second card, and 128 GB of RAM. The routed
27
  experts are split across GPU and RAM, and the ~195 GiB engram tables are memory-mapped and read from disk on demand,
28
  so they cost disk space, not RAM.
29
 
30
- Both quants were measured against the release-precision experts, not against another quant, and both answer all
31
- 16 retrieval and multi-hop questions exactly at depths up to 255K tokens.
32
 
33
  > **These files need a DeepSeek-V4.1 build of llama.cpp.** Upstream llama.cpp cannot load DeepSeek-V4.1 yet
34
  > ([#28696](https://github.com/ggml-org/llama.cpp/pull/28696) is open). See [Runtime](#runtime).
35
 
36
  ## Which one do I want?
37
 
38
- | | **IQ3_XXS** (recommended) | **IQ2_XS** |
39
- |---|---|---|
40
- | Weights you must hold in VRAM + RAM | 193.9 GiB | 171.7 GiB |
41
- | Engram tables (mmapped from disk) | 194.6 GiB | 194.6 GiB (identical files) |
42
- | Total download | 388.4 GiB | 366.3 GiB |
43
- | Mean KL divergence vs release experts | **0.141** | 0.183 |
44
- | Median KL divergence | **0.023** | 0.033 |
45
- | Same top-1 token | **88.5%** | 86.7% |
46
- | Perplexity vs reference | +5.2% | +10.4% |
47
- | Depth ladder (needle + multi-hop, to 255K) | **16/16** | **16/16** |
48
- | Decode, reference box (see below) | 18.1-20.6 tok/s (16 at 255K) | 20.4-22.9 tok/s (17 at 255K) |
49
- | Prefill, 32K-255K prompt | 475-527 tok/s | 529-592 tok/s |
50
 
51
  - **IQ3_XXS:** pick this if your VRAM + RAM can hold ~194 GiB of weights plus context.
52
  - **IQ2_XS:** pick this when you're ~20 GiB short. It costs 30% more KL divergence and ~1.8 points of top-1
53
  agreement. On the same box it decodes ~10% faster and prefills ~15% faster, because fewer expert bytes sit in RAM.
 
 
 
54
 
55
- Shards `-00006-`, `-00008-`, `-00009-` and `-00010-` (the two engram tables among them, 228.4 GiB together) are
56
- byte-identical between the two quants (see `SHA256SUMS`). If you want both quants, download those once and copy or
57
- link them under the second quant's name.
58
 
59
  ## Recipes
60
 
61
- | tensor group | IQ3_XXS | IQ2_XS |
62
- |---|---|---|
63
- | Routed experts gate/up (80 tensors) | `IQ3_XXS` on 28 layers, `IQ2_XS` on 12 | `IQ2_XS` |
64
- | Routed experts down (40) | `IQ3_XXS` | `IQ3_XXS` |
65
- | Shared expert, attention, indexer/engram projections | `Q8_0` (copied from the source, not re-quantized) | same |
66
- | Router, mHC, compressor, norms, some indexer/engram parts | `F32` / `BF16`, as in the source | same |
67
- | `token_embd`, `output` | `BF16` | `BF16` |
68
- | Engram tables (2 x 384M rows x 256) | `Q8_0`, see [Engram precision](#engram-precision) | same |
69
 
70
  The routed experts are about 95% of the non-engram bytes, so their bit width decides quality.
71
 
@@ -134,6 +137,11 @@ CUDA_VISIBLE_DEVICES=0,1 llama-server \
134
  (`-ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1'`). That places 77.3 GiB on GPU0 and 85.1 GiB in RAM;
135
  GPU0 holds 90,730 MiB after load and peaks at 90,940 MiB in the 255K rung.
136
 
 
 
 
 
 
137
  ### Settings that matter
138
 
139
  - **`-np 1`.** On this runtime, more than one cache stream scores tokens measurably differently from the
@@ -169,17 +177,18 @@ each (81,920 tokens), one sequence per batch.
169
  **Depth ladder.** Facts planted at 10/50/90% depth in a synthetic log, and four questions per rung: a verbatim
170
  needle, a single retrieval, a three-fact multi-hop sum, and a comparison. Thinking on, effort low. The IQ3_XXS
171
  rows were measured before the shard-1 metadata rewrite (same tensor bytes); the release files repeated the 32K rung, 4/4.
 
172
 
173
- | prompt tokens | IQ3_XXS | prefill | decode at that depth | IQ2_XS | prefill | decode at that depth |
174
- |---:|---|---:|---:|---|---:|---:|
175
- | 32,137 | 4/4 | 475 tok/s | 19.2-20.1 tok/s | 4/4 | 592 tok/s | 21.7-22.9 tok/s |
176
- | 64,141 | 4/4 | 527 tok/s | 19.2-20.6 tok/s | 4/4 | 591 tok/s | 20.7-22.9 tok/s |
177
- | 125,244 | 4/4 | 513 tok/s | 18.1-19.5 tok/s | 4/4 | 570 tok/s | 20.4-21.5 tok/s |
178
- | 255,340 | 4/4 | 479 tok/s | 15.6-16.1 tok/s | 4/4 | 529 tok/s | 16.3-17.3 tok/s |
179
 
180
  **Functional checks:** arithmetic, thinking open/close, tool calls with thinking on and off, and a tool-result
181
- round trip all pass on both quants, using the release files and their embedded template. Short-prompt decode is
182
- 18.4-18.9 tok/s (IQ3_XXS) and 21.4-22.4 tok/s (IQ2_XS).
183
 
184
  ## Calibration
185
 
@@ -199,6 +208,7 @@ sequences of 8,192 tokens. All 46,080 routed-expert slots received activations.
199
  |---|---|
200
  | IQ3_XXS | `DeepSeek-V4.1-Flash-1017-IQ3_XXS-000{01..10}-of-00010.gguf` |
201
  | IQ2_XS | `DeepSeek-V4.1-Flash-1017-IQ2_XS-000{01..10}-of-00010.gguf` |
 
202
  | imatrix | `imatrix/DeepSeek-V4.1-Flash-1017-imatrix.gguf` |
203
  | runtime patch | `patches/dsv41s-runtime.patch` |
204
  | checksums | `SHA256SUMS` (`sha256sum -c SHA256SUMS --ignore-missing`) |
 
22
 
23
  # DeepSeek-V4.1-Flash-1017 GGUF: imatrix quants for one big workstation
24
 
25
+ Three imatrix-guided quants of [DeepSeek-V4.1-Flash-1017](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017),
26
  built and validated on one machine: an RTX PRO 6000 96 GB, a 16 GB second card, and 128 GB of RAM. The routed
27
  experts are split across GPU and RAM, and the ~195 GiB engram tables are memory-mapped and read from disk on demand,
28
  so they cost disk space, not RAM.
29
 
30
+ All three quants were measured against the release-precision experts, not against another quant, and all three
31
+ answer all 16 retrieval and multi-hop questions exactly at depths up to 255K tokens.
32
 
33
  > **These files need a DeepSeek-V4.1 build of llama.cpp.** Upstream llama.cpp cannot load DeepSeek-V4.1 yet
34
  > ([#28696](https://github.com/ggml-org/llama.cpp/pull/28696) is open). See [Runtime](#runtime).
35
 
36
  ## Which one do I want?
37
 
38
+ | | **IQ3_XXS** (recommended) | **IQ2_XS** | **IQ2_XXS** |
39
+ |---|---|---|---|
40
+ | Weights you must hold in VRAM + RAM | 193.9 GiB | 171.7 GiB | 152.0 GiB |
41
+ | Engram tables (mmapped from disk) | 194.6 GiB | 194.6 GiB (identical files) | 194.6 GiB (identical files) |
42
+ | Total download | 388.4 GiB | 366.3 GiB | 346.5 GiB |
43
+ | Mean KL divergence vs release experts | **0.141** | 0.183 | 0.229 |
44
+ | Median KL divergence | **0.023** | 0.033 | 0.046 |
45
+ | Same top-1 token | **88.5%** | 86.7% | 85.1% |
46
+ | Perplexity vs reference | +5.2% | +10.4% | +14.1% |
47
+ | Depth ladder (needle + multi-hop, to 255K) | **16/16** | **16/16** | **16/16** |
48
+ | Decode, reference box (see below) | 18.1-20.6 tok/s (16 at 255K) | 20.4-22.9 tok/s (17 at 255K) | 22.1-25.9 tok/s (19 at 255K) |
49
+ | Prefill, 32K-255K prompt | 475-527 tok/s | 529-592 tok/s | 636-727 tok/s |
50
 
51
  - **IQ3_XXS:** pick this if your VRAM + RAM can hold ~194 GiB of weights plus context.
52
  - **IQ2_XS:** pick this when you're ~20 GiB short. It costs 30% more KL divergence and ~1.8 points of top-1
53
  agreement. On the same box it decodes ~10% faster and prefills ~15% faster, because fewer expert bytes sit in RAM.
54
+ - **IQ2_XXS:** pick this when you're ~40 GiB short of IQ3_XXS, or when speed matters more than fidelity. Against
55
+ IQ3_XXS it costs 62% more KL divergence and 3.4 points of top-1 agreement; on the same box it decodes ~25% faster
56
+ and prefills ~40% faster.
57
 
58
+ Shards `-00008-`, `-00009-` and `-00010-` (the two engram tables among them, 204.1 GiB together) are byte-identical
59
+ in all three quants, and `-00006-` is also shared by IQ3_XXS and IQ2_XS (see `SHA256SUMS`). If you want more than
60
+ one quant, download those once and copy or link them under the other quant's name.
61
 
62
  ## Recipes
63
 
64
+ | tensor group | IQ3_XXS | IQ2_XS | IQ2_XXS |
65
+ |---|---|---|---|
66
+ | Routed experts gate/up (80 tensors) | `IQ3_XXS` on 28 layers, `IQ2_XS` on 12 | `IQ2_XS` | `IQ2_XXS` |
67
+ | Routed experts down (40) | `IQ3_XXS` | `IQ3_XXS` | `Q2_K` |
68
+ | Shared expert, attention, indexer/engram projections | `Q8_0` (copied from the source, not re-quantized) | same | same |
69
+ | Router, mHC, compressor, norms, some indexer/engram parts | `F32` / `BF16`, as in the source | same | same |
70
+ | `token_embd`, `output` | `BF16` | `BF16` | `BF16` |
71
+ | Engram tables (2 x 384M rows x 256) | `Q8_0`, see [Engram precision](#engram-precision) | same | same |
72
 
73
  The routed experts are about 95% of the non-engram bytes, so their bit width decides quality.
74
 
 
137
  (`-ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1'`). That places 77.3 GiB on GPU0 and 85.1 GiB in RAM;
138
  GPU0 holds 90,730 MiB after load and peaks at 90,940 MiB in the 255K rung.
139
 
140
+ **IQ2_XXS on the same box:** use `-ncmoe 18` and send block 18's down-projection experts to RAM as well
141
+ (`-ot 'blk\.(20|21)\.ffn_(gate|up|down)_exps\.weight=CUDA1,blk\.18\.ffn_down_exps\.weight=CPU'`). That places
142
+ 78.1 GiB on GPU0, 7.1 GiB on GPU1 and 65.5 GiB in RAM; GPU0 holds 91,002 MiB after load and peaks at 91,180 MiB in
143
+ the 255K rung.
144
+
145
  ### Settings that matter
146
 
147
  - **`-np 1`.** On this runtime, more than one cache stream scores tokens measurably differently from the
 
177
  **Depth ladder.** Facts planted at 10/50/90% depth in a synthetic log, and four questions per rung: a verbatim
178
  needle, a single retrieval, a three-fact multi-hop sum, and a comparison. Thinking on, effort low. The IQ3_XXS
179
  rows were measured before the shard-1 metadata rewrite (same tensor bytes); the release files repeated the 32K rung, 4/4.
180
+ The IQ2_XS and IQ2_XXS rows are the release files. Every rung scored 4/4 on all three quants.
181
 
182
+ | prompt tokens | IQ3_XXS prefill / decode | IQ2_XS prefill / decode | IQ2_XXS prefill / decode |
183
+ |---:|---:|---:|---:|
184
+ | 32,137 | 475 / 19.2-20.1 tok/s | 592 / 21.7-22.9 tok/s | 727 / 24.2-25.1 tok/s |
185
+ | 64,141 | 527 / 19.2-20.6 tok/s | 591 / 20.7-22.9 tok/s | 720 / 23.9-25.9 tok/s |
186
+ | 125,244 | 513 / 18.1-19.5 tok/s | 570 / 20.4-21.5 tok/s | 688 / 22.1-23.8 tok/s |
187
+ | 255,340 | 479 / 15.6-16.1 tok/s | 529 / 16.3-17.3 tok/s | 636 / 18.8-19.4 tok/s |
188
 
189
  **Functional checks:** arithmetic, thinking open/close, tool calls with thinking on and off, and a tool-result
190
+ round trip all pass on all three quants, using the release files and their embedded template. Short-prompt decode
191
+ is 18.4-18.9 tok/s (IQ3_XXS), 21.4-22.4 tok/s (IQ2_XS) and 24.1-24.8 tok/s (IQ2_XXS).
192
 
193
  ## Calibration
194
 
 
208
  |---|---|
209
  | IQ3_XXS | `DeepSeek-V4.1-Flash-1017-IQ3_XXS-000{01..10}-of-00010.gguf` |
210
  | IQ2_XS | `DeepSeek-V4.1-Flash-1017-IQ2_XS-000{01..10}-of-00010.gguf` |
211
+ | IQ2_XXS | `DeepSeek-V4.1-Flash-1017-IQ2_XXS-000{01..10}-of-00010.gguf` |
212
  | imatrix | `imatrix/DeepSeek-V4.1-Flash-1017-imatrix.gguf` |
213
  | runtime patch | `patches/dsv41s-runtime.patch` |
214
  | checksums | `SHA256SUMS` (`sha256sum -c SHA256SUMS --ignore-missing`) |