--- license: apache-2.0 library_name: transformers pipeline_tag: image-text-to-text base_model: huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated base_model_relation: quantized tags: - ornith - abliterated - uncensored - w8a8 - int8 - compressed-tensors - llmcompressor - vllm - moe - multimodal - mtp - speculative-decoding language: - en - zh --- # Ornith-1.5-35B-A3B-abliterated-W8A8 INT8 W8A8 quantisation of huihui-ai's abliterated Ornith 1.5 35B-A3B, with the vision tower, the MoE routers and the MTP head deliberately left in bf16. **35.7 GiB**, `compressed-tensors` 0.18.0, format `int-quantized`. Architecture `Qwen3_5MoeForConditionalGeneration`: vision intact, MTP intact. ## Lineage Every link is pinned to the commit this artifact was actually built from. | step | source | revision | |---|---|---| | foundation | [`ornith-ai/Ornith-1.5-35B-A3B`](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B) | upstream release | | abliteration (tower) | [`huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated`](https://huggingface.co/huihui-ai/Huihui-Ornith-1.5-35B-A3B-abliterated) | `7269f0953054d68b3cacbd9ad729921258732007` | | MTP head | [`shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY`](https://huggingface.co/shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY) | `2b19b31bfe1659c6b0d9459ec3cbd87e34a322ef` | | quantisation | llmcompressor `QuantizationModifier`, recipe in [`recipe.yaml`](recipe.yaml) | this repo | **The MTP head is not the one the base model ships, and that is deliberate.** It is grafted from shisa-ai's distill. See "The MTP head" below, including a correction to a claim about the original head that circulates and is wrong. ## Why this abliteration Three abliterations of Ornith 1.5 35B-A3B are published. Only huihui-ai's keeps the model whole, and this quantisation preserves that: | | this repo | the other published W8A8 | the other abliterations | |---|---|---|---| | abliterated | yes | **no** | yes | | architecture | `Qwen3_5MoeForConditionalGeneration` | `Qwen3_5MoeForCausalLM` | one downgrades to `ForCausalLM` | | `model.visual.*` tensors | **333** | 0 | 0 | | MTP head | yes (grafted) | yes (original, unfused) | dropped | The two alternatives drop both the MTP head and the vision tower (64.6 GiB against the official 67.0 GiB, zero of each). A text-only checkpoint silently drops every image and OCR caller, which is a failure that looks like nothing at all: the model answers, it just cannot see. ## What was quantised, and what was not Scheme `W8A8`: **8-bit integer weights, symmetric, per channel**, and **8-bit integer activations, dynamic**. Targets `Linear`. No calibration data (data-free round-to-nearest), so nothing about this checkpoint is tuned to a calibration corpus. Left in bf16, on purpose: - **the whole vision tower**, all 27 blocks and the merger - **every MoE router**: `mlp.gate` and `shared_expert_gate` on all 40 layers. Routers pick experts, and an 8-bit router picks different experts - **the `linear_attn` modules and their norms**. Ornith 1.5 is a hybrid, and the linear-attention state is precision-sensitive - **`lm_head`** - **the entire MTP head** (`re:.*mtp.*`) ## The MTP head The head here is shisa-ai's distill, not the one Ornith ships. Three reasons, and the first thing to say is what is *not* a reason. **The original head is trained. The widespread claim that it is not is false.** The claim (from the model's own discussion #10) is that every projection has a standard deviation of exactly 0.0200, i.e. `initializer_range`. Probing the published safetensors directly on 2026-08-27 does not support it: `mtp.norm.weight` has mean **+1.9251**, `mtp.layers.0.post_attention_layernorm.weight` **+0.8686**, and the projection standard deviations vary by a factor of three (`mtp.fc` 0.0085, `shared_expert.gate_proj` 0.0089, `mlp.gate` 0.0096, `o_proj` 0.0185, `q_proj` 0.0247). A single init at `initializer_range` would put all of them at 0.0200 and leave the norms near 0.02. For scale, the same checkpoint's trained `model.language_model.norm.weight` is 1.6397 against the head's 1.9251. **Please do not repeat the claim.** The actual reasons: 1. **It accepts nearly twice as many drafted tokens.** Measured in vLLM by shisa-ai and published alongside the head: Ornith's own head accepts 37.20% (mean accepted run 2.116), a zero-shot graft of the Qwen3.6 head 50.19% (2.506), and this head, a 12K KL distill over two mixed-data epochs warm-started from Ornith's own, **69.27% (3.078)**. 2. **It transfers.** It is a fine-tune rather than a retrain: element-wise against the official head the deltas are 0.5% to 8.7% of each tensor's range. A head only has to be accepted by the tower it drafts for, and this one is within a few percent of the head that already pairs with it. 3. **Layout.** Ornith ships the head's 256 experts *unfused*, as 785 separate `mtp.layers.0.mlp.experts.N.*` tensors. This head ships the **19-tensor fused form** (`experts.gate_up_proj` `[256, 1024, 2048]`, `experts.down_proj` `[256, 2048, 512]`). DFlash was the other candidate and was rejected: its acceptance falls from 89.8% at 8K context to 48.8% near 256K, it is *slower* than plain autoregressive decode at 252K/8K (0.67x), and it reportedly breaks multimodal. ## Why W8A8 rather than FP8 or NVFP4 This matters on Ampere and not much anywhere else. vLLM's W8A8 INT8 scheme declares a minimum compute capability of **75**, so an A40 (8.6) multiplies in its own INT8 tensor cores. The FP8 W8A8 scheme declares **89**, so below Lovelace vLLM resolves the checkpoint to the weight-only scheme and serves it through Marlin: the activations go back to 16 bits and the memory saving is real while the speedup is not. NVFP4 has no native FP4 path on Ampere at all. So on Ada, Hopper and Blackwell, prefer the official FP8 build. On Ampere, this one. ## Serving ```bash vllm serve nuoram/Ornith-1.5-35B-A3B-abliterated-W8A8 \ --tensor-parallel-size 2 \ --enable-auto-tool-choice --tool-call-parser qwen3_xml ``` Fits 2x 48GB (A40, A6000) with room for a long context. ## Credit and licence The model is ornith-ai's, the abliteration is huihui-ai's, and the MTP head is shisa-ai's. This repository changes the numeric format and nothing else. Licence follows the upstream Ornith release; if that differs from the Apache-2.0 declared here, upstream governs. Abliterated models have had refusal behaviour removed. You are responsible for what you do with it.