--- license: other license_name: swift-open-license-1.0 license_link: https://huggingface.co/ukisai/Swift-Qwen3.8-27b library_name: gguf pipeline_tag: image-text-to-text base_model: - ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP base_model_relation: quantized tags: - gguf - llama.cpp - qwen3_8 - abliterated - uncensored - unsloth - imatrix - dynamic-quant - mtp --- # Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF GGUF quants of [ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP](https://huggingface.co/ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP), an abliterated [Swift-Qwen3.8-27B](https://huggingface.co/ukisai/Swift-Qwen3.8-27b) (UkisAI's reasoning-efficient fine-tune of [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)). For vLLM and SGLang: [NVFP4](https://huggingface.co/ajgazin/Swift-Qwen3.8-27B-Uncensored-NVFP4). The Swift 1.5 version is [Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF](https://huggingface.co/ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF). - **Uncensored:** 15/100 refusals against Swift's 98/100, KL divergence 0.0634. - **Unsloth Dynamic 3.0 layout:** every tensor has the type Unsloth chose for its quant of the same size, with Unsloth's importance matrix. - **MTP head included** in every main GGUF, for self-speculative decoding in llama.cpp. - **Vision:** image and video input through the included projector. - **Measured:** KL divergence against the BF16 (see [Quality](#quality)). ## Quants | Quant | Size | Use | | :-- | :-- | :-- | | UD-Q2_K_XL | 9.2 GiB | Fits a 12 GB GPU. | | UD-IQ3_XXS | 10.2 GiB | Also fits a 12 GB GPU. Prefer it to UD-Q2_K_XL if it fits: clearly closer to the BF16. | | UD-IQ3_S | 11.2 GiB | Fits a 16 GB GPU with more context room than UD-Q3_K_XL. | | UD-Q3_K_XL | 12.2 GiB | Fits a 16 GB GPU. | | UD-IQ4_XS | 13.3 GiB | Smallest 4-bit. Fits a 16 GB GPU, with less context room than UD-Q3_K_XL. | | UD-Q4_K_S | 14.3 GiB | Smaller 4-bit. Fits a 20 GB GPU, or a 16 GB GPU with short context. | | UD-Q4_K_XL | 16.4 GiB | Fits a 24 GB GPU with room for long context. | | UD-Q5_K_S | 17.4 GiB | Fits a 24 GB GPU with more context room than UD-Q5_K_M. | | UD-Q5_K_M | 18.4 GiB | Fits a 24 GB GPU. | | UD-Q6_K_XL | 23.6 GiB | Best fit for a 32 GB GPU. | | UD-Q8_K_XL | 29.3 GiB | Near-lossless. Fits a 48 GB GPU; on a 32 GB GPU it needs partial CPU offload. | | BF16 | 50.9 GiB | Unquantized. The source of the quants, and a reference for measuring them or making other sizes. | Each quant is one file, `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-.gguf`; the BF16 is `Swift-Qwen3.8-27B-Uncensored-MTP-BF16.gguf`. Other files: | File | Size | Use | | :-- | :-- | :-- | | `mmproj-BF16.gguf` | 0.9 GiB | Vision projector, needed only for image and video input. | | `tensor_types.tsv` | 42 KB | Per-tensor quantization types for each size (see [Quantization](#quantization)). | | `kld_results.csv` | 5 KB | Every statistic behind the [Quality](#quality) table. | ## Quality Each quant compared with the BF16 it was made from, token by token: wikitext-2 test set, 36 × 8192 tokens, f16 KV cache, llama.cpp `94659b076`. Lower KLD and higher top-1 are better. ![Mean KL divergence of each quant against file size](kld_vs_size.png) | Size | GiB | Mean KLD | 99 % KLD | Same top-1 | | :-- | --: | --: | --: | --: | | UD-Q2_K_XL | 9.2 | 0.1796 | 1.99 | 85.2 % | | UD-IQ3_XXS | 10.2 | 0.1255 | 1.20 | 87.9 % | | UD-IQ3_S | 11.2 | — | — | — | | UD-Q3_K_XL | 12.2 | 0.0726 | 0.58 | 91.8 % | | UD-IQ4_XS | 13.3 | — | — | — | | UD-Q4_K_S | 14.3 | 0.0521 | 0.45 | 94.1 % | | UD-Q4_K_XL | 16.4 | 0.0324 | 0.16 | 95.8 % | | UD-Q5_K_S | 17.4 | 0.0272 | 0.11 | 96.4 % | | UD-Q5_K_M | 18.4 | 0.0207 | 0.080 | 96.9 % | | UD-Q6_K_XL | 23.6 | 0.0116 | 0.028 | 98.2 % | | UD-Q8_K_XL | 29.3 | 0.0060 | 0.012 | 98.8 % | - **UD-IQ3_S and UD-IQ4_XS** were added after this measurement run, so they have no numbers yet and are absent from the chart for the same reason. Both come with the next batch. - **Mean KLD** is the average divergence of the quant's next-token distribution from the BF16's. **99 % KLD** is the level only the worst 1 % of tokens exceed. **Same top-1** is how often both pick the same most likely token. - Quality follows file size with no exceptions. The knee is at UD-Q5_K_M: above it a GiB buys little, below it each GiB saved costs about twice as much. - The typical token is barely touched (median KLD is 0.0002 at UD-Q8_K_XL and 0.003 at UD-Q4_K_XL); the mean is carried by a small share of tokens that change a lot. From UD-Q3_K_XL down the error also turns one-sided, taking 0.4–1.3 % of probability from the BF16's token on average. This measures quantization only, against this repository's own BF16. It is a different quantity from the 0.0634 under [The model](#the-model), which is the abliteration measured against Swift. Prose only: code and chat-formatted text were not measured. Every statistic is in `kld_results.csv`. It also includes perplexity, which does not rank these files: several quants score below the BF16. ## Usage (llama.cpp) ```bash # text only llama-server -m Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q6_K_XL.gguf -ngl 99 -c 32768 # with vision (image and video input) llama-server -m Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q6_K_XL.gguf -ngl 99 -c 32768 \ --mmproj mmproj-BF16.gguf # self-speculative decoding with the built-in MTP head llama-server -m Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q6_K_XL.gguf -ngl 99 -c 32768 \ --spec-type draft-mtp ``` `--spec-type draft-mtp` needs a llama.cpp build with MTP support for `qwen35`. The MTP head loads from the main GGUF; there is no separate draft file. Sampling, as for Swift and Qwen: temperature 1.0, top_p 0.95, top_k 20, min_p 0. The model thinks before answering by default. ## The model The source applies the refusal direction of [orcarouter/Qwen3.8-27B-Uncensored](https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored) (Arditi et al. 2024, one direction) to Swift's weights: 131 tensors edited (attention outputs, `mlp.down_proj`, `embed_tokens`, MTP layer included), everything else Swift's. Method, direction and scripts are on the [source model's card](https://huggingface.co/ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP). | Model | Refusals | KL divergence | | :-- | :-: | :-: | | **BF16 source** (against Swift) | **15/100** | **0.0634** | | Swift-Qwen3.8-27B | 98/100 | 0 | | *Reference:* orcarouter/Qwen3.8-27B-Uncensored (against Qwen3.8-27B) | 17/100 | 0.0621 | Measured with [Heretic](https://github.com/p-e-w/heretic) on the BF16 weights (100 prompts from `mlabonne/harmful_behaviors`, keyword refusal detector; first-token KL on `mlabonne/harmless_alpaca`; thinking skipped). Not re-measured on the quants. ## Quantization - **Conversion:** the source model's BF16 safetensors (all 1199 tensors, MTP included) were converted with llama.cpp's `convert_hf_to_gguf.py`, once for the language model and once with `--mmproj` for the vision projector. - **Quantization:** `llama-quantize` with Unsloth's `imatrix_unsloth.gguf` from [unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF), plus a `--tensor-type-file` that sets every tensor's type to the one in Unsloth's GGUF of the same size. | Size | Tensor types (F32 norms etc. excluded) | | :-- | :-- | | UD-Q2_K_XL | 112×IQ3_XXS, 98×Q8_0, 67×IQ2_S, 57×IQ3_S, 48×IQ2_XXS, 34×IQ2_XS, 21×Q4_K, 20×IQ1_S, 19×IQ4_XS, 16×Q2_K, 6×Q6_K, 5×Q3_K, 2×Q5_K, 1×IQ1_M | | UD-IQ3_XXS | 120×IQ3_XXS, 106×IQ3_S, 98×Q8_0, 45×IQ4_XS, 35×IQ2_S, 26×Q4_K, 24×IQ2_XXS, 12×IQ2_XS, 10×Q2_K, 8×IQ1_S, 7×Q5_K, 6×Q3_K, 6×Q6_K, 3×IQ1_M | | UD-IQ3_S | 127×IQ3_S, 98×Q8_0, 88×IQ4_XS, 77×IQ3_XXS, 23×Q4_K, 21×IQ2_S, 16×Q5_K, 16×Q3_K, 12×IQ2_XS, 12×IQ2_XXS, 7×Q6_K, 6×Q2_K, 2×IQ1_S, 1×IQ4_NL | | UD-Q3_K_XL | 156×IQ4_XS, 111×IQ3_S, 98×Q8_0, 36×Q4_K, 34×IQ3_XXS, 26×Q5_K, 15×IQ2_S, 12×Q3_K, 7×Q6_K, 4×IQ2_XS, 3×Q2_K, 2×IQ4_NL, 2×IQ2_XXS | | UD-IQ4_XS | 211×IQ4_XS, 98×Q8_0, 55×Q4_K, 51×Q5_K, 46×IQ3_S, 17×IQ3_XXS, 9×Q3_K, 8×Q6_K, 7×IQ2_S, 2×IQ2_XS, 1×IQ4_NL, 1×Q2_K | | UD-Q4_K_S | 172×IQ4_XS, 99×Q8_0, 95×Q4_K, 80×Q5_K, 18×Q6_K, 15×IQ3_S, 13×Q3_K, 7×IQ4_NL, 5×IQ3_XXS, 1×IQ2_S, 1×IQ2_XS | | UD-Q4_K_XL | 191×Q5_K, 110×Q8_0, 70×IQ4_XS, 69×Q4_K, 56×Q6_K, 6×IQ4_NL, 3×Q3_K, 1×IQ3_S | | UD-Q5_K_S | 232×Q5_K, 117×Q8_0, 84×Q6_K, 35×Q4_K, 32×IQ4_XS, 4×IQ4_NL, 2×Q3_K | | UD-Q5_K_M | 189×Q5_K, 160×Q6_K, 124×Q8_0, 19×IQ4_XS, 12×Q4_K, 2×IQ4_NL | | UD-Q6_K_XL | 310×Q8_0, 168×Q6_K, 27×Q5_K, 1×Q4_K | | UD-Q8_K_XL | 453×Q8_0, 53×BF16 | The imatrix was calibrated on the original Qwen3.8-27B; Swift's low-rank fine-tune and the rank-one abliteration change the weights only slightly. ### Checks - **Layout:** each quant's 866 tensors have exactly the types and shapes of Unsloth's file of the same size. - **MTP:** every main GGUF has 65 blocks, with the MTP layer's 15 tensors in `blk.64` (`nextn_predict_layers = 1`). llama.cpp's `--spec-type draft-mtp` loads it and drafts with it. - **Tokenizer and chat template:** - The vocabulary, merges and special tokens are the same as in Unsloth's Qwen3.8-27B GGUF. - `add_bos_token = false` and the padding token `<|endoftext|>` come from Swift's own tokenizer config. - The embedded chat template is byte-identical to Swift's, which is Qwen3.8-27B's. - **Vision projector:** all 334 tensors are identical to Qwen3.8-27B's vision projector. Neither Swift's fine-tune nor the abliteration touches the vision tower. - **Load test:** every quant loads and runs on a 32 GB GPU: fully offloaded up to UD-Q6_K_XL, with 56 of 65 layers offloaded for UD-Q8_K_XL. Each produced sane logits on a short perplexity run before upload; the measurement that ranks them is under [Quality](#quality). ### Reproduce ```bash # 1. convert (Swift-Qwen3.8-27B-Uncensored-MTP/ is a download of the source repository) python convert_hf_to_gguf.py Swift-Qwen3.8-27B-Uncensored-MTP --outtype bf16 \ --model-name Swift-Qwen3.8-27B-Uncensored-MTP --outfile Swift-Qwen3.8-27B-Uncensored-MTP-BF16.gguf python convert_hf_to_gguf.py Swift-Qwen3.8-27B-Uncensored-MTP --outtype bf16 --mmproj \ --model-name Swift-Qwen3.8-27B-Uncensored-MTP --outfile mmproj-BF16.gguf # 2. per-size --tensor-type-file from tensor_types.tsv (here UD-Q5_K_M), one anchored regex per tensor awk -F'\t' 'NR==1 { for (i=2; i<=NF; i++) if ($i=="UD-Q5_K_M") c=i; next } { n=$1; gsub(/\./,"\\\\.",n); print "^" n "$=" $c }' \ tensor_types.tsv > tensor_types_UD-Q5_K_M.txt # 3. quantize. The last argument is the fallback type for tensors the file doesn't list: # UD-Q2_K_XL Q2_K, UD-IQ3_XXS IQ3_XXS, UD-IQ3_S IQ3_S, UD-Q3_K_XL Q3_K_L, # UD-IQ4_XS IQ4_XS, UD-Q4_K_S Q4_K_S, UD-Q4_K_XL Q4_K_M, UD-Q5_K_S Q5_K_S, # UD-Q5_K_M Q5_K_M, UD-Q6_K_XL Q6_K, UD-Q8_K_XL Q8_0 llama-quantize --imatrix imatrix_unsloth.gguf \ --tensor-type-file tensor_types_UD-Q5_K_M.txt \ Swift-Qwen3.8-27B-Uncensored-MTP-BF16.gguf \ Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q5_K_M.gguf Q5_K_M ``` `tensor_types.tsv` holds the type of every non-F32 tensor in each of Unsloth's sizes, one row per tensor and one column per size. The regexes must be anchored (`^…$`). `llama-quantize` matches them unanchored, so a bare `output\.weight` line would also match every `attn_output.weight`. ## Not evaluated General benchmarks, KL divergence of the quants on code or chat-formatted text, refusal behaviour in thinking mode and on the quants, whether Swift's shorter reasoning traces survive, and MTP acceptance against Swift. ## License Derivative of Swift-Qwen3.8-27B, under the **Swift Open License v1.0** ([Swift model card](https://huggingface.co/ukisai/Swift-Qwen3.8-27b)): free for individuals and organizations up to US$1,000,000 annual recurring revenue, above that commercial use needs a Swift Enterprise License from UkisAI. Qwen3.8-27B and orcarouter/Qwen3.8-27B-Uncensored are Apache 2.0. ## Intended use The model answers requests the original declines. You are responsible for how you use it and for complying with applicable law and the license. ## Credits - [Qwen](https://huggingface.co/Qwen) for Qwen3.8-27B. - [UkisAI](https://ukisai.com) for Swift-Qwen3.8-27B. - [OrcaRouter](https://huggingface.co/orcarouter) for Qwen3.8-27B-Uncensored and its refusal direction. - [Unsloth](https://huggingface.co/unsloth) for the Dynamic 3.0 quantization layout and imatrix. - Arditi et al., [Refusal in Language Models Is Mediated by a Single Direction](https://arxiv.org/abs/2406.11717) (2024). - [Heretic](https://github.com/p-e-w/heretic), used for evaluation.