--- license: other license_name: swift-open-license-1.0 license_link: https://huggingface.co/ukisai/Swift-Qwen3.8-27b library_name: gguf pipeline_tag: image-text-to-text base_model: - ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP base_model_relation: quantized tags: - gguf - llama.cpp - qwen3_8 - abliterated - uncensored - unsloth - imatrix - dynamic-quant - mtp --- # Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF GGUF quants of [ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP](https://huggingface.co/ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP), an abliterated [Swift-Qwen3.8-27B](https://huggingface.co/ukisai/Swift-Qwen3.8-27b) (UkisAI's reasoning-efficient fine-tune of [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)). For vLLM and SGLang: [NVFP4](https://huggingface.co/ajgazin/Swift-Qwen3.8-27B-Uncensored-NVFP4). - **Uncensored:** 15/100 refusals against Swift's 98/100, KL divergence 0.0634. - **Unsloth Dynamic 3.0 layout:** every tensor has the type Unsloth chose for its quant of the same size, with Unsloth's importance matrix. - **MTP head included** in every main GGUF, for self-speculative decoding in llama.cpp. - **Vision:** image and video input through the included projector. ## Files | File | Size | Use | | :-- | :-- | :-- | | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q2_K_XL.gguf` | 9.2 GiB | Fits a 12 GB GPU. | | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q3_K_XL.gguf` | 12.2 GiB | Fits a 16 GB GPU. | | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q4_K_XL.gguf` | 16.4 GiB | Fits a 24 GB GPU with room for long context. | | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q5_K_S.gguf` | 17.4 GiB | Fits a 24 GB GPU with more context room than UD-Q5_K_M. | | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q5_K_M.gguf` | 18.4 GiB | Fits a 24 GB GPU. | | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q6_K_XL.gguf` | 23.6 GiB | Best fit for a 32 GB GPU. | | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 29.3 GiB | Near-lossless. Fits a 48 GB GPU; on a 32 GB GPU it needs partial CPU offload. | | `Swift-Qwen3.8-27B-Uncensored-MTP-BF16.gguf` | 50.9 GiB | Unquantized. The source of the quants, and a reference for measuring them or making other sizes. | | `mmproj-BF16.gguf` | 0.9 GiB | Vision projector, needed only for image and video input. | | `tensor_types.tsv` | 30 KB | Per-tensor quantization types for each size (see [Quantization](#quantization)). | ## Usage (llama.cpp) ```bash # text only llama-server -m Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q6_K_XL.gguf -ngl 99 -c 32768 # with vision (image and video input) llama-server -m Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q6_K_XL.gguf -ngl 99 -c 32768 \ --mmproj mmproj-BF16.gguf # self-speculative decoding with the built-in MTP head llama-server -m Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q6_K_XL.gguf -ngl 99 -c 32768 \ --spec-type draft-mtp ``` `--spec-type draft-mtp` needs a llama.cpp build with MTP support for `qwen35`. The MTP head loads from the main GGUF; there is no separate draft file. Sampling, as for Swift and Qwen: temperature 1.0, top_p 0.95, top_k 20, min_p 0. The model thinks before answering by default. ## The model The source applies the refusal direction of [orcarouter/Qwen3.8-27B-Uncensored](https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored) (Arditi et al. 2024, one direction) to Swift's weights: 131 tensors edited (attention outputs, `mlp.down_proj`, `embed_tokens`, MTP layer included), everything else Swift's. Method, direction and scripts are on the [source model's card](https://huggingface.co/ajgazin/Swift-Qwen3.8-27B-Uncensored-MTP). | Model | Refusals | KL divergence | | :-- | :-: | :-: | | **BF16 source** (against Swift) | **15/100** | **0.0634** | | Swift-Qwen3.8-27B | 98/100 | 0 | | *Reference:* orcarouter/Qwen3.8-27B-Uncensored (against Qwen3.8-27B) | 17/100 | 0.0621 | Measured with [Heretic](https://github.com/p-e-w/heretic) on the BF16 weights (100 prompts from `mlabonne/harmful_behaviors`, keyword refusal detector; first-token KL on `mlabonne/harmless_alpaca`; thinking skipped). Not re-measured on the quants. ## Quantization - **Conversion:** the source model's BF16 safetensors (all 1199 tensors, MTP included) were converted with llama.cpp's `convert_hf_to_gguf.py`, once for the language model and once with `--mmproj` for the vision projector. - **Quantization:** `llama-quantize` with Unsloth's `imatrix_unsloth.gguf` from [unsloth/Qwen3.8-27B-GGUF](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF), plus a `--tensor-type-file` that sets every tensor's type to the one in Unsloth's GGUF of the same size. | Size | Tensor types (F32 norms etc. excluded) | | :-- | :-- | | UD-Q2_K_XL | 112×IQ3_XXS, 98×Q8_0, 67×IQ2_S, 57×IQ3_S, 48×IQ2_XXS, 34×IQ2_XS, 21×Q4_K, 20×IQ1_S, 19×IQ4_XS, 16×Q2_K, 6×Q6_K, 5×Q3_K, 2×Q5_K, 1×IQ1_M | | UD-Q3_K_XL | 156×IQ4_XS, 111×IQ3_S, 98×Q8_0, 36×Q4_K, 34×IQ3_XXS, 26×Q5_K, 15×IQ2_S, 12×Q3_K, 7×Q6_K, 4×IQ2_XS, 3×Q2_K, 2×IQ4_NL, 2×IQ2_XXS | | UD-Q4_K_XL | 191×Q5_K, 110×Q8_0, 70×IQ4_XS, 69×Q4_K, 56×Q6_K, 6×IQ4_NL, 3×Q3_K, 1×IQ3_S | | UD-Q5_K_S | 232×Q5_K, 117×Q8_0, 84×Q6_K, 35×Q4_K, 32×IQ4_XS, 4×IQ4_NL, 2×Q3_K | | UD-Q5_K_M | 189×Q5_K, 160×Q6_K, 124×Q8_0, 19×IQ4_XS, 12×Q4_K, 2×IQ4_NL | | UD-Q6_K_XL | 310×Q8_0, 168×Q6_K, 27×Q5_K, 1×Q4_K | | UD-Q8_K_XL | 453×Q8_0, 53×BF16 | The imatrix was calibrated on the original Qwen3.8-27B; Swift's low-rank fine-tune and the rank-one abliteration change the weights only slightly. ### Checks - **Layout:** each quant's 866 tensors have exactly the types and shapes of Unsloth's file of the same size. - **MTP:** every main GGUF has 65 blocks, with the MTP layer's 15 tensors in `blk.64` (`nextn_predict_layers = 1`). llama.cpp's `--spec-type draft-mtp` loads it and drafts with it. - **Tokenizer and chat template:** - The vocabulary, merges and special tokens are the same as in Unsloth's Qwen3.8-27B GGUF. - `add_bos_token = false` and the padding token `<|endoftext|>` come from Swift's own tokenizer config. - The embedded chat template is byte-identical to Swift's, which is Qwen3.8-27B's. - **Vision projector:** all 334 tensors are identical to Qwen3.8-27B's vision projector. Neither Swift's fine-tune nor the abliteration touches the vision tower. - **Load test:** every quant loads and runs on a 32 GB GPU: fully offloaded up to UD-Q6_K_XL, with 56 of 65 layers offloaded for UD-Q8_K_XL. Perplexity on a short wikitext-2 sample (8 × 2048 tokens, about ±0.13): | Size | UD-Q2_K_XL | UD-Q3_K_XL | UD-Q4_K_XL | UD-Q5_K_M | UD-Q6_K_XL | UD-Q8_K_XL | | :-- | --: | --: | --: | --: | --: | --: | | PPL | 5.35 | 5.12 | 5.08 | 5.10 | 5.09 | 5.08 | The sample is too small to rank the sizes above Q2. ### Reproduce ```bash # 1. convert (Swift-Qwen3.8-27B-Uncensored-MTP/ is a download of the source repository) python convert_hf_to_gguf.py Swift-Qwen3.8-27B-Uncensored-MTP --outtype bf16 \ --model-name Swift-Qwen3.8-27B-Uncensored-MTP --outfile Swift-Qwen3.8-27B-Uncensored-MTP-BF16.gguf python convert_hf_to_gguf.py Swift-Qwen3.8-27B-Uncensored-MTP --outtype bf16 --mmproj \ --model-name Swift-Qwen3.8-27B-Uncensored-MTP --outfile mmproj-BF16.gguf # 2. per-size --tensor-type-file from tensor_types.tsv (here UD-Q5_K_M), one anchored regex per tensor awk -F'\t' 'NR==1 { for (i=2; i<=NF; i++) if ($i=="UD-Q5_K_M") c=i; next } { n=$1; gsub(/\./,"\\\\.",n); print "^" n "$=" $c }' \ tensor_types.tsv > tensor_types_UD-Q5_K_M.txt # 3. quantize. The last argument is the fallback type for tensors the file doesn't list: # UD-Q2_K_XL Q2_K, UD-Q3_K_XL Q3_K_L, UD-Q4_K_XL Q4_K_M, UD-Q5_K_S Q5_K_S, UD-Q5_K_M Q5_K_M, # UD-Q6_K_XL Q6_K, UD-Q8_K_XL Q8_0 llama-quantize --imatrix imatrix_unsloth.gguf \ --tensor-type-file tensor_types_UD-Q5_K_M.txt \ Swift-Qwen3.8-27B-Uncensored-MTP-BF16.gguf \ Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q5_K_M.gguf Q5_K_M ``` `tensor_types.tsv` holds the type of every non-F32 tensor in each of Unsloth's sizes, one row per tensor and one column per size. The regexes must be anchored (`^…$`). `llama-quantize` matches them unanchored, so a bare `output\.weight` line would also match every `attn_output.weight`. ## Not evaluated KL divergence of each quant against BF16, general benchmarks, refusal behaviour in thinking mode, whether Swift's shorter reasoning traces survive, and MTP acceptance against Swift. ## License Derivative of Swift-Qwen3.8-27B, under the **Swift Open License v1.0** ([Swift model card](https://huggingface.co/ukisai/Swift-Qwen3.8-27b)): free for individuals and organizations up to US$1,000,000 annual recurring revenue, above that commercial use needs a Swift Enterprise License from UkisAI. Qwen3.8-27B and orcarouter/Qwen3.8-27B-Uncensored are Apache 2.0. ## Intended use The model answers requests the original declines. You are responsible for how you use it and for complying with applicable law and the license. ## Credits - [Qwen](https://huggingface.co/Qwen) for Qwen3.8-27B. - [UkisAI](https://ukisai.com) for Swift-Qwen3.8-27B. - [OrcaRouter](https://huggingface.co/orcarouter) for Qwen3.8-27B-Uncensored and its refusal direction. - [Unsloth](https://huggingface.co/unsloth) for the Dynamic 3.0 quantization layout and imatrix. - Arditi et al., [Refusal in Language Models Is Mediated by a Single Direction](https://arxiv.org/abs/2406.11717) (2024). - [Heretic](https://github.com/p-e-w/heretic), used for evaluation.