Quality: corpus strided PPL replaces low-power top-1 probe; spec-table harness footnotes; chart updated

README.md

@@ -17,22 +17,22 @@ tags:
 
 # Qwen3.8-27B-Alis-MLX-6bit
 
-![Build size vs decode speed across the 8-bit, 6-bit and 4-bit builds, with Korean top-1 agreement and both speculative paths, MTP k=2 and the DSpark drafter](assets/builds.png)
+![Build size vs decode speed across the 8-bit, 6-bit and 4-bit builds, with Korean corpus perplexity and both speculative paths, MTP k=2 and the DSpark drafter](assets/builds.png)
 
-A 6-bit MLX quantization of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for Apple silicon, with the **vision tower and the MTP head both preserved**. 21.5 GB on disk, 27.3 tok/s decode, and top-1 agreement with the bf16 original of 97.3 / 96.8 / 100 on Korean / English / code. This is the balanced build of the set: it fits a 32 GB Mac and keeps non-English output intact.
+A 6-bit MLX quantization of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) for Apple silicon, with the **vision tower and the MTP head both preserved**. 21.5 GB on disk, 27.3 tok/s decode, Korean corpus perplexity **statistically indistinguishable from the bf16 original** (6.1018 vs 6.0954, paired over 35K tokens), and only hairline — though measurable — excesses on English and code. This is the balanced build of the set: it fits a 32 GB Mac and keeps non-English output intact.
 
 ## At a glance
 
-| Build | Size (GB) | Decode (tok/s) | Prefill (tok/s) | Peak RAM (GB) | Korean NLL | top-1 ko / en / code |
-|---|---|---|---|---|---|---|
-| bf16 reference | 51.8 | 12.6 | 410 | 51.1 | 1.661 | 100 / 100 / 100 |
-| 8-bit | 27.9 | 21.8 | 429 | 28.2 | 1.663 | 99.1 / 98.4 / 100 |
-| 6-bit (this repo) | 21.5 | 27.3 | 424 | 21.9 | 1.679 | 97.3 / 96.8 / 100 |
-| 4-bit | 15.2 | 37.5 | 436 | 15.6 | 1.679 | 85.7 / 93.5 / 100 |
+| Build | Size (GB) | Decode (tok/s) | Prefill (tok/s) | Peak RAM (GB) | PPL en / ko / code |
+|---|---|---|---|---|---|
+| bf16 reference | 51.8 | 12.6 | 410 | 51.1 | 5.7734 / 6.0954 / 1.6813 |
+| 8-bit | 27.9 | 21.8 | 429 | 28.2 | 5.7760 / 6.0987 / 1.6815 |
+| 6-bit (this repo) | 21.5 | 27.3 | 424 | 21.9 | 5.7924 / 6.1018 / 1.6854 |
+| 4-bit (AWQ) | 15.2 | 37.5 | 436 | 15.6 | 5.8450 / 6.2609 / 1.8105 |
 
-Measured on one Apple M3 Ultra (512 GB unified memory), all builds queued back to back in a single batch so they share machine state. Peak RAM is the load-plus-probe peak at short context; add KV cache for your own context length (see below).
+Speed and RAM were measured on one Apple M3 Ultra (512 GB unified memory), all builds queued back to back in a single batch so they share machine state; the 4-bit row of that batch was that repo's earlier uniform release, and its AWQ replacement is the same size with plain decode re-measured at 37.7 tok/s. Peak RAM is the load-plus-probe peak at short context; add KV cache for your own context length (see below). Perplexity is corpus-scale strided PPL — ctx 2048, stride 512 — described in the quality section.
 
-Against bf16 this build is 42% of the size at 2.17× the decode speed. Against the 8-bit build it is 23% smaller and 25% faster, for 1.8 points of Korean top-1 agreement.
+Against bf16 this build is 42% of the size at 2.17× the decode speed. Against the 8-bit build it is 23% smaller and 25% faster, and the paired corpus test prices that step at +0.28% PPL on English and +0.23% on code — both significant, both hairline — with Korean not separable at all.
 
 ## The vision tower is intact
 
@@ -74,13 +74,13 @@ python -m mlx_vlm.generate \
   --prompt "Describe every shape in this image, with its color and position."
 ```
 
-Verified end to end before publishing: I fed a hand-drawn test image (red circle top left, blue square top right, green triangle at the bottom) to the 4-bit build of this set and it described every color, shape, and position correctly. The vision path is the same bf16 tower in all three builds.
+Verified end to end before publishing: I fed a hand-drawn test image (red circle top left, blue square top right, green triangle at the bottom) to the 4-bit build of this set — its earlier uniform release — and it described every color, shape, and position correctly. The vision path is the same bf16 tower in all three builds.
 
 ## Which build should I use?
 
-- **6-bit (this repo, 21.5 GB)** — the default I would reach for first. It keeps Korean top-1 agreement at 97.3, which is where the 4-bit build falls apart, while staying 6.4 GB smaller and 25% faster than 8-bit. Peak was 21.9 GB at short context and it leaves usable headroom on a 32 GB Mac, which is the machine this build exists for.
-- **8-bit (27.9 GB)** — the fidelity choice. Take it when the output is the product and the machine is 48 GB or larger: Korean NLL 0.002 above bf16, top-1 99.1 on Korean. Also the right reference to compare a smaller build against.
-- **4-bit (15.2 GB)** — the reach build, and the fastest at 37.5 tok/s. Take it when the model would otherwise not fit, or for code and English work. Its Korean top-1 agreement falls to 85.7, an 11.6-point drop from this build, which is what the extra 6.3 GB here buys.
+- **6-bit (this repo, 21.5 GB)** — the default I would reach for first. Its Korean corpus PPL is statistically indistinguishable from bf16 — the axis where the 4-bit build pays its clearest user-visible cost — while staying 6.4 GB smaller and 25% faster than 8-bit. Peak was 21.9 GB at short context and it leaves usable headroom on a 32 GB Mac, which is the machine this build exists for.
+- **8-bit (27.9 GB)** — the fidelity choice. Take it when the output is the product and the machine is 48 GB or larger: statistically indistinguishable from bf16 on every corpus slice. Also the right reference to compare a smaller build against.
+- **4-bit (15.2 GB, now the AWQ recipe)** — the reach build, and the fastest at 37.5 tok/s. Take it when the model would otherwise not fit. Its corpus PPL runs +1.2% / +2.7% / +7.7% over bf16 on English / Korean / code — the one build whose cost is significant on every slice, and the Korean share of it is what the extra 6.3 GB here buys back.
 
 All three carry the same vision tower and the same MTP head. The choice is purely language-model precision.
 
@@ -148,7 +148,7 @@ mlx_lm.generate --model avlp12/Qwen3.8-27B-Alis-MLX-6bit \
   --max-tokens 512
 ```
 
-**Every speculative figure below was measured on the 4-bit build of this set, not on this one.** It is quoted here because it is what the fork's two speculative paths do on this checkpoint, and because the ordering between them is informative — but the multiples do not transfer to this build, for a reason spelled out under the table. Measured on one Apple M3 Ultra (512 GB), greedy, 240 tokens per prompt, timed after an 8-token warmup, every configuration in the same process, over four prompts: chat, code, math, and Korean.
+**Every speculative figure below was measured on the 4-bit build of this set — its earlier uniform 4-bit weights — not on this one.** It is quoted here because it is what the fork's two speculative paths do on this checkpoint, and because the ordering between them is informative — but the multiples do not transfer to this build, for a reason spelled out under the table. Measured on one Apple M3 Ultra (512 GB), greedy, 240 tokens per prompt, timed after an 8-token warmup, every configuration in the same process, over four prompts: chat, code, math, and Korean.
 
 | Configuration, 4-bit build | 4-prompt avg | 3 English prompts | chat | code | math | ko |
 |---|---|---|---|---|---|---|
@@ -158,6 +158,8 @@ mlx_lm.generate --model avlp12/Qwen3.8-27B-Alis-MLX-6bit \
 
 Lossless on that build: all four prompts reproduced plain greedy decoding **token for token through 80 tokens**.
 
+**Three footnotes on this table, added after an audit of the measurement harness.** First, the harness did **not stop at end-of-sequence** — every run decoded a fixed 240 tokens, so on prompts whose answer ends early, part of the window measures post-termination behavior, and per-prompt figures are sensitive to what a build happens to do after its answer ends. Read the averages, not the per-prompt cells. Second, acceptance-rate comparisons **between builds** are only meaningful on a common token sequence, scored paired — each build free-running its own text measures the text as much as the build. Third, the 4-bit build measured here was that repo's earlier uniform release; the AWQ weights now hosted there measure **within ±1%** of it on both speculative paths under an EOS-cut paired protocol. A full re-measurement under the corrected harness is a separate track.
+
 **Why this build cannot inherit those multiples.** What lifted them is a small-M quantized-GEMM kernel that the fork now enables from inside `mlx_lm.utils.load()`. It fires only in the multi-token verification window (M = 6..8) and only on **4-bit, group-64** weights, so it never engages at 6 bits; plain decode at M = 1 is untouched on every build either way. The gap it fills is upstream: [ml-explore/mlx#4265](https://github.com/ml-explore/mlx/issues/4265). Set `MLXLM_NO_FAST_QMM=1` to opt out.
 
 **This 6-bit build has never been measured under either speculative path** — the MTP head is present, the fork drives it, and the drafter below accepts it as a target, but I have no number for any of the three rows on this build. Start at k=2 and measure your own workload rather than interpolating from a table taken on a different precision.
@@ -212,7 +214,7 @@ Speculative decoding pays off in proportion to how predictable the continuation
 
 The hybrid layout is why long context is cheap on this model. Only the **16 full-attention layers** hold a growing KV cache; the 48 GatedDeltaNet layers carry a fixed-size recurrent state instead. With 4 KV heads at head_dim 256, that comes to **64 KB per token** in bf16 across the whole model. That per-token figure is a property of the layout and is identical in all three builds.
 
-Measured at 16K context on the 4-bit and 8-bit builds, as whole-process peaks — so they include activations and prefill buffers, not the cache alone. This 6-bit build was not measured; it sits between the two columns.
+Measured at 16K context on the 4-bit and 8-bit builds (the 4-bit column on that repo's earlier uniform release), as whole-process peaks — so they include activations and prefill buffers, not the cache alone. This 6-bit build was not measured; it sits between the two columns.
 
 | KV precision | Peak RAM (GB), 4-bit build | Peak RAM (GB), 8-bit build | top-1 agreement |
 |---|---|---|---|
@@ -249,31 +251,33 @@ This is a flat, data-free round-to-nearest recipe: no calibration set, no activa
 
 ## Quality
 
-Teacher-forced against the bf16 original on three held-out passages — Korean prose (112 tokens), English prose (62 tokens), and Python (91 tokens) — in a single pass, reporting **top-1 agreement**: the fraction of positions where this build's argmax matches bf16's argmax on identical context. Korean NLL is the mean negative log-likelihood on that same Korean passage.
+Corpus-scale strided perplexity against fixed token streams: window 2048, stride 512, so every scored token sees up to 2048 tokens of left context. Three slices — **English** = wikitext-2 test (35,981 tokens), **Korean** = Korean Wikipedia (35,179 tokens), **code** = CPython standard library (32,295 tokens). Every build scores the same tokens in the same order, so build-to-build comparisons are paired per token, and significance is judged on the paired excess NLL with a blockwise standard error (512-token blocks) — roughly 16–20× the sensitivity of the probe this section used to carry.
 
-| Slice | bf16 | 8-bit | 6-bit (this repo) | 4-bit |
-|---|---|---|---|---|
-| Korean top-1 | 100 | 99.1 | 97.3 | 85.7 |
-| English top-1 | 100 | 98.4 | 96.8 | 93.5 |
-| Code top-1 | 100 | 100 | 100 | 100 |
-| Korean NLL | 1.661 | 1.663 | 1.679 | 1.679 |
+| Build | en · wikitext-2 | ko · Korean wiki | code · CPython |
+|---|---|---|---|
+| bf16 reference | 5.7734 | 6.0954 | 1.6813 |
+| 8-bit | 5.7760 | 6.0987 | 1.6815 |
+| **6-bit (this repo)** | **5.7924** | **6.1018** | **1.6854** |
+| 4-bit (AWQ) | 5.8450 | 6.2609 | 1.8105 |
 
-This build is where the size curve and the quality curve are still on speaking terms. Korean and English both stay in the high 90s, and the gap to 8-bit is 1.8 and 1.6 points — small enough that on most work the 6.4 GB and the 25% decode speed are the better trade. The gap to 4-bit, by contrast, is 11.6 points on Korean, which is not a trade at all; it is a different quality class.
+This build is where the size curve and the quality curve are still on speaking terms. Against bf16, paired: **Korean is not separable at all** — the 95% CI on this build's Korean excess NLL includes zero — while English and code carry excesses the test can find but you will struggle to feel: +0.0033 and +0.0024 nats/token (+0.33% and +0.24% PPL). The 8-bit build is indistinguishable from bf16 on all three slices; the 4-bit AWQ build is significantly above on all three, at +1.2% / +2.7% / +7.7% — on Korean and code, a different quality class from this one.
 
 Three honest caveats:
 
-- **Code being flat at 100 across every build does not prove 6-bit is lossless.** Code is the most predictable of the three slices; it failed to separate any build and therefore carries no evidence about them.
-- **top-1 agreement is a strict positional metric, not a score.** A mismatch is not automatically a worse token, and it does not translate into a percentage of wrong output. It is a sensitive damage detector, used here to locate damage rather than to price it.
-- **This build and the 4-bit build record an identical Korean NLL of 1.679** while their top-1 agreement differs by 11.6 points. Two readings fit that: the 4-bit damage is concentrated at near-tie positions where the ranking flips but the probability mass barely moves, or a 112-token passage is simply too short for NLL to separate them. I did not run further probes to decide between the two, so do not read the NLL row as evidence that this build and the 4-bit one are equivalent — the metric that separates them says otherwise.
+- **"Indistinguishable" means the test could not separate them at this sample size, not that they are identical.** A still larger corpus might resolve a Korean difference; 35K paired tokens did not.
+- **Perplexity prices average next-token surprise, not task success.** It ranks the builds with real statistical power, but it is not a benchmark suite.
+- **The hairline English and code excesses are significant but tiny.** Treat them as the honest price of this tier, not as a reason to step up to 8-bit — step up when you need the reference build, not because of 0.3%.
+
+**Why this table changed.** Earlier revisions of this card ranked the builds with a top-1 agreement probe over three short passages (62–112 tokens). At that size the probe had no statistical power: its standard error ran several times the effects it claimed to rank, and on English it read the sign of a recipe difference **backwards**. It is removed rather than restated.
 
 ## Limitations and caveats
 
-- **Probe-scale measurement, not a benchmark suite.** Quality comes from a single teacher-forced pass over three short passages (112 / 62 / 91 tokens), greedy argmax.
-- **No standard benchmarks were run.** There are no MMLU, GSM8K, or HumanEval numbers here, and none should be inferred from the agreement table.
+- **Perplexity is the only quality metric here.** It is corpus-scale (≈103K scored tokens, paired per token) and it replaced a probe that had no statistical power — but it is still average next-token surprise, not a task suite.
+- **No standard benchmarks were run.** There are no MMLU, GSM8K, or HumanEval numbers here, and none should be inferred from the perplexity table.
 - **Vision quality is unquantified.** The tower is preserved bit-exact and verified to work on a shapes probe. That is a functional check, not a VQA evaluation. No image benchmark was run at any bit-width.
 - **MTP and KV-cache figures come from the neighboring builds.** This build was not measured on either axis; the head and the layout are present and identical, the numbers are not.
-- **Shape-dependent numerics.** The hybrid stack takes different kernel paths depending on the shape of the input it receives — a batched teacher-forced pass is not the same shape as incremental decode. On the same prefix, positions whose top-2 logits are nearly tied can flip between those paths. Agreement figures carry a run-to-run wobble at this precision; do not read the last decimal as stable.
-- **Long-form reasoning is unprobed.** Every probe here is short, while low-bit damage tends to concentrate in long reasoning chains. Spot-check your own thinking-mode prompts.
+- **Shape-dependent numerics.** The hybrid stack takes different kernel paths depending on the shape of the input it receives — the batched scoring pass behind the perplexity table is not the same shape as incremental decode, and near-tie positions can resolve differently between those paths. Perplexity is far less sensitive to this than an argmax metric, but do not expect bit-identical numbers from a different harness.
+- **Long-form generated reasoning is unprobed.** Perplexity scores 2048-token windows of existing text; it does not exercise long generated reasoning chains, where low-bit damage tends to concentrate. Spot-check your own thinking-mode prompts.
 
 ## License and attribution
 

assets/make_builds_png.py

@@ -136,35 +136,27 @@ txt(200, 331, "50.36 tok/s · 1.34× vs plain 4-bit", 13.5, CM)
 
 # ============ 4-bit ============
 ax.plot([179.1], [407.5], "o", markersize=16 * PT, color=C4, zorder=6)
-txt(200, 366, "4-bit", 15.5, C4, "bold")
+txt(200, 366, "4-bit AWQ", 15.5, C4, "bold")
 txt(200, 385, "15.2 GB · 37.5 tok/s plain", 13.5, C4)
-txt(200, 403, "ko top-1 85.7%", 12.5, C4, "bold")
-bar(200, 411, 84, GREY, 0.16, z=4)
-bar(200, 411, 23.9, C4)
+txt(200, 403, "ko PPL 6.261 · +2.7% vs bf16", 12.5, C4, "bold")
 
 # ============ 6-bit (label to the left) ============
 ax.plot([275.0], [478.9], "o", markersize=16 * PT, color=C6, zorder=6)
 txt(257, 482, "6-bit", 15.5, C6, "bold", ha="right")
 txt(257, 501, "21.5 GB · 27.3 tok/s", 13.5, C6, ha="right")
-txt(257, 519, "ko top-1 97.3%", 12.5, C6, ha="right")
-bar(173, 527, 84, GREY, 0.16, z=4)
-bar(173, 527, 72.7, C6)
+txt(257, 519, "ko PPL 6.102 (≈bf16)", 12.5, C6, ha="right")
 
 # ============ 8-bit ============
 ax.plot([372.4], [517.4], "o", markersize=16 * PT, color=C8, zorder=6)
 txt(390, 459, "8-bit", 15.5, C8, "bold")
 txt(390, 478, "27.9 GB · 21.8 tok/s", 13.5, C8)
-txt(390, 496, "ko top-1 99.1%", 12.5, C8)
-bar(390, 504, 84, GREY, 0.16, z=4)
-bar(390, 504, 80.2, C8)
+txt(390, 496, "ko PPL 6.099 (≈bf16)", 12.5, C8)
 
 # ============ bf16 ============
 ax.plot([736.1], [581.8], "o", markersize=16 * PT, color=CB, zorder=6)
 txt(790, 512, "bf16 reference", 15.5, CB, "bold", ha="right")
 txt(790, 531, "51.8 GB · 12.6 tok/s", 13.5, CB, ha="right")
-txt(790, 549, "ko top-1 100%", 12.5, CB, ha="right")
-bar(706, 557, 84, GREY, 0.16, z=4)
-bar(706, 557, 84, CB)
+txt(790, 549, "ko PPL 6.095 — reference", 12.5, CB, ha="right")
 
 # ============ in-plot callout: the Korean caveat ============
 ax.add_patch(FancyBboxPatch((459.5, 190.5), 0.1, 15,
@@ -206,18 +198,18 @@ txt(858, 556, "DSpark drafter is a separate 0.76 GB download.", 13, MUT)
 
 # ============ footnotes ============
 for fy, s in [
-    (682, "Measured on one M3 Ultra 512GB box, queued back-to-back. Bars show "
-          "Korean top-1 agreement vs bf16 on an 80–100% scale; the probe is a "
-          "teacher-forced single pass over"),
-    (701, "three passages (ko 112 / en 62 / code 91 tokens). At 4-bit, English "
-          "is 93.5% and code 100%, and Korean is 11.6 points behind 6-bit. "
-          "No standard benchmarks were run."),
+    (682, "Measured on one M3 Ultra 512GB box, queued back-to-back. Quality = "
+          "corpus strided perplexity — ctx 2048, stride 512, paired per token "
+          "vs bf16 — over en wikitext-2 35,981, ko Korean"),
+    (701, "wiki 35,179, code CPython 32,295 tokens. 8-bit ≈ bf16 on every "
+          "slice, 6-bit ≈ bf16 on Korean; 4-bit AWQ: +1.2% en, +2.7% ko, "
+          "+7.7% code. No standard benchmarks were run."),
     (720, "Speculative decoding: greedy, 240 tokens per prompt, timed after an "
-          "8-token warmup, every configuration in one process on this 4-bit "
-          "build. Both paths are lossless — through"),
-    (739, "80 tokens each reproduced plain greedy decoding token for token. "
-          "MTP is the checkpoint's own vendor multi-token-prediction head at "
-          "k=2; DSpark drives an external 1.36B drafter."),
+          "8-token warmup, in one process on the 4-bit repo's earlier uniform "
+          "weights (the AWQ swap measures within ±1%,"),
+    (739, "EOS-cut paired). Both lossless — each reproduced plain greedy "
+          "decoding token for token through 80 tokens. MTP is the vendor "
+          "multi-token head at k=2; DSpark drives an external 1.36B drafter."),
     (758, "Plotted values are the 4-prompt average — single prompts overstate "
           "speculation: DSpark reads 75.1 tok/s on code and 91.5 on math "
           "against that 62.2 average.")]: