Release 0.2 card: version up front, comparison table, QAT, compute

States release 0.2 and the release-0.1 tag at the top, what changed, the comparison with Unsloth UD-IQ3_XXS, DT-IQ3_XXS, Mirai S and release 1 on the same evaluation, QAT wording and the G4 compute. inventory.json updated for the documents. Weights unchanged.

MODIFICATIONS.md

@@ -4,7 +4,7 @@
 
 ## Modified files
 
-- **`export/release_vq3pA_mixh_s600_k1_mat/` (release 0.2) and `export/mix25in_mixr_lr64mix/` (first release), each with `layer_NN.safetensors`, `layer_NN_mixer.safetensors`, `lm_head.safetensors`:** the text model's linear weights replaced by quantized representations: GPTQ lookup-table indices and codebooks (vector 2x64, vector 2x16 and scalar LUT4), per-channel scales, per-channel INT8 for the full-attention K / V projections, online Hadamard rotation seeds for the MLP basis and, in release 0.2, for the token-mixer projections (weights stored in the rotated basis), and rank-64 FP16 residual factors for the token-mixer projections. Recipe: [docs/QUANTIZATION.md](https://github.com/Anemll/anemll-forge/blob/main/docs/QUANTIZATION.md). Release 0.2 stores every MLP matrix as `vector 2x64 + pcs` (three bits per weight) and is fitted by GPTQ with the residual factors inside the sequential pass, then by quantization-aware distillation to the BF16 model that trains the lookup-table values, scales, factors and LM head with every index frozen.
+- **`export/release_vq3pA_mixh_s600_k1_mat/` (release 0.2) and `export/mix25in_mixr_lr64mix/` (first release), each with `layer_NN.safetensors`, `layer_NN_mixer.safetensors`, `lm_head.safetensors`:** the text model's linear weights replaced by quantized representations: GPTQ lookup-table indices and codebooks (vector 2x64, vector 2x16 and scalar LUT4), per-channel scales, per-channel INT8 for the full-attention K / V projections, online Hadamard rotation seeds for the MLP basis and, in release 0.2, for the token-mixer projections (weights stored in the rotated basis), and rank-64 FP16 residual factors for the token-mixer projections. Recipe: [docs/QUANTIZATION.md](https://github.com/Anemll/anemll-forge/blob/main/docs/QUANTIZATION.md). Release 0.2 stores every MLP matrix as `vector 2x64 + pcs` (three bits per weight) and is fitted by GPTQ with the residual factors inside the sequential pass, then by quantization-aware training (QAT) against the BF16 model that trains the lookup-table values, scales, factors and LM head with every index frozen.
 
 - **`drafter/drafter_quant.safetensors`:** the linear weights of [ProCreations/Ternary-Bonsai-2-27B-DFlash2 at `4cfb6ad03268fed0f60ca96c1a659c0b1c77e50b`](https://huggingface.co/ProCreations/Ternary-Bonsai-2-27B-DFlash2/tree/4cfb6ad03268fed0f60ca96c1a659c0b1c77e50b) (Apache-2.0, [drafter/LICENSE](drafter/LICENSE)) replaced by LUT4 GPTQ indices, codebooks and scales, calibrated on the `q7_cal` traces.
 

README.md

@@ -17,6 +17,15 @@ tags:
 
 # ANEMLL · Qwen3.8-27B, quantized for Core AI export
 
+**Current version: release 0.2** (tag [`release-0.2`](https://huggingface.co/anemll/anemll-quantized-qwen3.8-27b-for-CoreAI/tree/release-0.2), export `release_vq3pA_mixh_s600_k1_mat`). The first release's export stays in `export/mix25in_mixr_lr64mix` (tag [`release-0.1`](https://huggingface.co/anemll/anemll-quantized-qwen3.8-27b-for-CoreAI/tree/release-0.1)).
+
+**What changed in release 0.2:**
+
+- **Three-bit MLP:** every MLP matrix (192) is a three-bit vector lookup table (`vector 2x64 + pcs`), where the first release used two-bit tables in 38 layers and four-bit tables in 26.
+- **Token-mixer rotations:** the DeltaNet and attention projections read inputs rotated online by 1,024-wide Hadamard blocks, as the MLP already did; the weights are stored in the rotated basis.
+- **QAT:** after GPTQ, 600 steps of quantization-aware training against the BF16 model (about 4 GPU-hours in total on one RTX PRO 6000).
+- **Result:** KL to BF16 0.054 against 0.151 (PyTorch, chats), 0.052 against 0.184 on the M6 ANE; top-1 agreement 91.9% against 86.0%. Size 9.44 GiB against 9.07 GiB. Comparison with other quantizations [below](#quality).
+
 **The quantized weights behind [ANEMLL Forge](https://github.com/Anemll/anemll-forge)'s Qwen3.8-27B for the Apple Neural Engine.** This repository holds the quantized target exports `release_vq3pA_mixh_s600_k1_mat` (release 0.2, the current packages) and `mix25in_mixr_lr64mix` (the first release), the quantized export of the paired DFlash2 speculative drafter, and the small parts of both original checkpoints the converter needs, so the published Core AI packages ([anemll/anemll-forge-qwen3.8-27B](https://huggingface.co/anemll/anemll-forge-qwen3.8-27B)), target and drafter, can be rebuilt, checked or varied without the original checkpoints (52 GB and 3.6 GB). It is a research artifact for conversion: the files hold lookup-table indices, codebooks and scales in ANEMLL's own layout, not a checkpoint that transformers or other runtimes can load directly.
 
 To run the model, use the prepared Core AI bundle: [anemll/anemll-forge-qwen3.8-27B](https://huggingface.co/anemll/anemll-forge-qwen3.8-27B) and the [ANEMLL Forge README](https://github.com/Anemll/anemll-forge). To rebuild those packages from this repository, follow [EXPORT.md](EXPORT.md).
@@ -47,7 +56,7 @@ Mixed precision with GPTQ, per weight matrix ([recipe and math](https://github.c
 - **Token mixers:** vector LUTs (`vector 2x16 + pcs`) in layers 0 to 23 and scalar LUT4 in layers 24 to 63; full-attention K / V projection weights use per-channel INT8. Rank-64 FP16 residual factors correct the DeltaNet and attention projections. **Release 0.2 adds online Hadamard rotations** to the projections' inputs (1,024-wide blocks, seeds `3000 + layer` for the readers and `4000 + layer` for `out_proj` / `o_proj`); the weights are stored in the rotated basis and the factors in the original one.
 - **Output head:** scalar LUT4 with per-output-channel scaling. Embeddings stay FP16 and are looked up on the host (the inference bundle's `model/embed_tokens_fp16.npy`).
 
-**Release 0.2 fitting** ([plan](https://github.com/Anemll/anemll-forge/blob/main/configs/quantization/mix25in_vq3pA.json)), in two stages. GPTQ fits the rank-64 factors inside the sequential pass (each layer quantizes what its factors leave), with 128 calibration rows (WikiText, BF16 chat traces with thinking, rendered agentic coding sessions). Then 600 steps of quantization-aware distillation train the lookup-table values, per-channel scales, factors and LM head to match BF16's next-token distribution (top-256 plus tail), with every index frozen, on those rows plus about 640 rows of BF16 responses (thinking included) to new prompts across code, agentic tool use, multilingual text, math and rare tokens. The training rows were checked against the evaluation sets (13-gram overlap 0.04 to 0.12% of evaluation windows, generic phrasing).
+**Release 0.2 fitting** ([plan](https://github.com/Anemll/anemll-forge/blob/main/configs/quantization/mix25in_vq3pA.json)), in two stages. GPTQ fits the rank-64 factors inside the sequential pass (each layer quantizes what its factors leave), with 128 calibration rows (WikiText, BF16 chat traces with thinking, rendered agentic coding sessions). Then 600 steps of quantization-aware training (QAT, by distillation: the loss is the KL divergence from BF16's next-token distribution, top-256 plus tail) train the lookup-table values, per-channel scales, factors and LM head with every index frozen, on those rows plus about 640 rows of BF16 responses (thinking included) to new prompts across code, agentic tool use, multilingual text, math and rare tokens. The training rows were checked against the evaluation sets (13-gram overlap 0.04 to 0.12% of evaluation windows, generic phrasing). On one Colab G4 (NVIDIA RTX PRO 6000 Blackwell, 96 GB): GPTQ 54 min, QAT 3.1 h.
 
 "2-bit" or "3-bit model" is an incomplete description: activations, recurrent state and the KV cache stay in floating point (the Core AI graph adds an INT8 value cache and 8-bit attention arithmetic of its own).
 
@@ -68,7 +77,19 @@ Results from other Macs (chip, memory, context, prefill and decode tok/s) are we
 
 ## Quality
 
-Teacher-forced KL divergence against the BF16 model, on a short trace (64 sequences, 40,023 positions), for the Core AI targets built from these exports on the M6 ANE: release 0.2 mean KL **0.052**, top-1 agreement **91.9%**, perplexity 2.131 (M5 Max: 0.052); first release 0.184, 86.0%, 2.414; the teacher's perplexity is 2.136. This is not a coding, reasoning, retrieval or long-context evaluation.
+Teacher-forced KL divergence against the BF16 model, the same evaluation for every model: 64 chats (40,023 positions; "dev", with the shared 45-token system prompt excluded unless marked full), held-out chats (37,076 positions), WikiText (16 x 1,024 tokens) and the last 2K tokens of 16 windows of 16,384 tokens from 12 long sessions. GGUF files were scored with llama.cpp on an M3 Ultra (the BF16 GGUF reproduces the PyTorch reference within 0.0004), Mirai S decoded to BF16 and scored in PyTorch, ANEMLL in PyTorch.
+
+| Model | Size (GiB, bpw) | dev KL (excl. prompt) | held-out KL | dev KL, full | top-1 | p99 | WikiText KL | 16K windows, last 2K |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| **ANEMLL release 0.2** | 9.44 (3.17)* | **0.0536** | **0.0565** | **0.0515** | **91.9%**† | **0.50**† | 0.0707 | **0.762** |
+| [Unsloth UD-IQ3_XXS](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) | 9.46 (3.17) | 0.0619 | 0.0651 | 0.1625 | 89.4% | 3.34 | **0.0506** | 1.108 |
+| [DT-IQ3_XXS](https://huggingface.co/drawthingsai/Qwen3.8-27B-GGUF) | 8.94 (3.00) | 0.0789 | 0.0833 | 0.0763 | 90.4% | 0.75 | 0.1246 | 1.204 |
+| Mirai S (2.4-bit trellis) | 7.56 (2.54) | 0.1226 | 0.1279 | 0.2716 | 85.1% | 5.73 | 0.1674 | not run |
+| ANEMLL release 1 | 9.07 (3.04)* | 0.1512 | 0.1617 | 0.1852 | 86.1% | 2.25 | not run | 1.514 |
+
+\* Size counts the quantized linear weights and the head, excluding the embedding (and MTP for the GGUF files); ANEMLL adds 0.30 GiB of rank-64 FP16 factors. † From the ANE run of the same trace.
+
+Compiled Core AI targets on the ANE reproduce the PyTorch KL: release 0.2 KL-512 **0.052** on the M6 and on the M5 Max, top-1 agreement **91.9%**, perplexity 2.131 (the teacher's 2.136); the first release 0.184, 86.0%, 2.414. UD-IQ3_XXS and Mirai S carry most of their full-trace KL on the shared system prompt, which BF16 predicts almost deterministically. KL measures closeness to BF16 on these texts; it is not a coding, reasoning or retrieval benchmark, and the GGUF models run on the GPU, not the ANE.
 
 ## Reproducibility
 

inventory.json

@@ -36,8 +36,8 @@
   },
   {
    "path": "MODIFICATIONS.md",
-   "bytes": 3113,
-   "sha256": "149484618bb54a35420565da4d7ad16e74b991ef67772377ede014cbd55fbd38"
+   "bytes": 3120,
+   "sha256": "a3f95fd2daa18cd04e76e117b19d517cbca93dbab36bd9da08882240577c96b9"
   },
   {
    "path": "NOTICE",
@@ -51,8 +51,8 @@
   },
   {
    "path": "README.md",
-   "bytes": 11770,
-   "sha256": "bc8bcf899f53d3714e583ace7ab0963a9a3ee22beac92e56afe175a1af1c3b36"
+   "bytes": 14679,
+   "sha256": "a94d7b4c9e3b6ca702b1ab411bccf50722ffd4f1f088fd614c074f80d0821a40"
   },
   {
    "path": "config.json",