Instructions to use NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- NInfer
How to use NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer with NInfer:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2
English | 简体中文
NInfer v3 artifacts (.ninfer) of the abliterated Qwen3.8-27B. The text tower, token embedding,
output head and MTP head are the blocks of the GSQ-RCO mixed-precision IQ2-class GGUF
(Underdog-Saluki Swift 1.5), stored byte for byte; the Vision tower and the DFlash2 adapter are
re-encoded from their own sources. Each file is self-contained: weights, tokenizer, chat template,
generation config, image and video preprocessor configs, logical bindings. The container is not a GGUF
file: transformers, llama.cpp and GGUF runtimes cannot read it, an NInfer v3 engine loads it.
Lineage
| Step | Source |
|---|---|
| Abliteration and GSQ-RCO IQ2 quantization | huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF — the text tower, token embedding, output head, MTP head and Vision tower these files store |
| DFlash2 drafter | z-lab/Qwen3.8-27B-DFlash2 — the @@-dflash2@@ file only |
| Ancestral base | Qwen/Qwen3.8-27B, Apache-2.0 |
| This repository | re-encodes those weights into NInfer v3 containers; nothing is retrained or re-quantized here |
base_model_relation: merge records the same relation in the card metadata.
Files
| File | Bytes | GiB | Components | SHA-256 |
|---|---|---|---|---|
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer |
8,850,451,200 | 8.24 | text, vision, mtp, proposal | c7295fd5ed1adc8371cb7290265cdc50037569c21df98acb101377bd0451a4f9 |
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer |
11,077,268,736 | 10.32 | text, vision, mtp, dflash2, proposal | 785e7b58cd3cdbfd07e7216ec5fb70647c238d623bd60697221eb9fd20a59df3 |
File B is file A plus the DFlash2 drafter (+2.13 GiB). Both carry the same text tower, Vision tower, MTP head and proposal head, so download one.
Download
hf download NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer \
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer --local-dir models
hf download NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer \
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer --local-dir models
Verify
sha256sum -c SHA256SUMS
The repository also carries each artifact's .conversion.json report, recording the source, method
and format of every object (1194 objects in file A: 1188 tensors and 6 resources; 1260 in file B).
What is inside
| Component | Assembled from | Representation | Parameters | Stored |
|---|---|---|---|---|
| text: the 64 layers, token embedding and output head | the GSQ-RCO GGUF's own blocks | gguf_blocks_v1 — each tensor keeps the ggml block type the quantizer assigned it (IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, Q2_K, Q4_K, Q6_K, Q8_0) and is copied row for row, byte for byte; nothing is re-quantized |
26.896 B | 7.361 GiB |
| vision | the release's BF16 mmproj, re-encoded | q4_g64_fp16, q5_g64_fp16, q6_g64_fp16, q8_g32_fp16 |
0.461 B | 0.275 GiB |
| mtp | the checkpoint's own blk.64.nextn.* tensors |
Q6_K blocks | 0.425 B | 0.324 GiB |
| proposal head | the output head's own rows, gathered | IQ3_S, the 131,072 most frequent tokens of NInfer's ranking | 0.671 B | 0.269 GiB |
| dflash2 (file B) | z-lab's Qwen3.8-27B-DFlash2, re-encoded | q8_g32_fp16 projections, bf16 parameters |
1.977 B | 2.126 GiB |
| total, file A | 28.45 B | 8.24 GiB | ||
| total, file B | 30.43 B | 10.32 GiB |
GDN controls (ssm_alpha, ssm_beta) are stored as BF16; norms, convolution, A_log and
dt_bias are exact, with the exporter's conventions restored (w instead of 1 + w, A_log
from -exp(A_log)). The GDN output projection keeps llama.cpp's tiled value-head order as an
input_columns Use auxiliary.
7.905 GiB of file A (96.1%) is stored in gguf_blocks_v1; the rest is the Vision tower in NInfer's
Vision formats and the BF16/F32 parameter tensors. File B adds the DFlash2 adapter, which is not part
of the GGUF either.
The model
| Text tower | 64 layers: 48 GDN linear-attention and 16 full-attention layers, every fourth |
| Attention | 24 query heads, 4 KV heads, head_dim 256; the output gate is fused into the query projection; partial RoPE 0.25, mRoPE sections [11, 11, 10], theta 1e7 |
| GDN | 16 key heads × 128, 48 value heads × 128, convolution kernel 4, state in FP32 |
| Widths | hidden 5120, intermediate 17408, vocabulary 248,320, native window 262,144 tokens |
| Vision tower | 27 blocks, hidden 1152, intermediate 4304, 16 heads, patch 16, temporal patch 2, merge 2, 2,304 positions |
| Chat template | NInfer's pinned qwen3_8.jinja: thinking on by default, effort xhigh, closed-turn reasoning retained |
Running
An NInfer v3 engine is required: the master branch of iamwavecut/ninfer-all, built, or
the container image. Kernels are selected from the architecture, configuration and bindings stored in
the artifact, so no flag declares the quantization.
Container
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
-v "$PWD/models:/models" -v ninfer-cache:/cache \
ghcr.io/iamwavecut/ninfer-all serve /models/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer \
--model-id huihui-qwen3.8-27b-iq2 --max-context 262144 --kv-capacity auto --kv-dtype int8 \
--gdn-state-fp16 --spec mtp --draft-tokens 4 --adaptive-mtp --lm-head-draft \
--vision --vision-residency overlay --vision-max-merged 1024
A build
File A drafts with the MTP head:
ninfer-serve Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer \
--model-id huihui-qwen3.8-27b-iq2 --host 127.0.0.1 --port 1145 \
--max-context 262144 --kv-capacity auto --kv-dtype int8 --gdn-state-fp16 \
--spec mtp --draft-tokens 4 --adaptive-mtp --lm-head-draft \
--vision --vision-residency resident --vision-max-merged 1024 \
--prefill-cublas
File B drafts with the DFlash2 adapter:
ninfer-serve Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer \
--model-id huihui-qwen3.8-27b-iq2-dflash2 --host 127.0.0.1 --port 1145 \
--max-context 32768 --kv-capacity auto --kv-dtype rk8v4 --gdn-state-fp16 \
--spec dflash2 --draft-tokens 5 --lm-head-draft \
--vision --vision-residency overlay --vision-max-merged 1024
The server answers OpenAI, Anthropic and Responses requests under /v1; a build listens on
127.0.0.1:8080 unless --host and --port say otherwise.
Flags
| Flag | Effect |
|---|---|
--kv-dtype |
KV storage: bf16, int8, fp8, nvfp4, k8v4, rk8v4, rk4v4, rk4v4-e8, rk2v4-e8 |
--kv-capacity N|auto |
shared KV capacity; auto takes what is free after the weights |
--vision-residency |
resident (device), overlay (device memory borrowed per image), cpu |
--vision-max-merged |
merged tokens of one image, 64..16384 |
--lm-head-draft |
draft through the proposal head instead of the full output head, and load it |
--gdn-state-fp16 |
keep the GDN recurrent state in FP16 |
--spec, --draft-tokens |
speculative backend and drafts per round, 1..15 |
Memory
File A loads 8.24 GiB of weights and file B 10.32 GiB. Both ran on a 16 GB laptop GPU with
--kv-capacity auto; file B leaves the smaller KV cache on the same card. The Vision tower adds
0.275 GiB of weights, resident or borrowed, and video is not supported.
Engine compatibility
| Capability | Requirement |
|---|---|
| File A — MTP drafting | any NInfer v3 build with --spec mtp |
| File B — DFlash2 drafting | a build that accepts --spec dflash2; an engine limited to MTP refuses the flag at startup and must run file A |
| Vision | text and image, in both files |
| Proposal head | --lm-head-draft |
On Windows the runtime path, model path and model name must be ASCII.
Conversion
Official recipe qwen3_8_27b_gguf, from the model's own blocks:
python3 -m tools.convert \
--model /path/to/Qwen3.8-27B \ # config.json + tokenizer files only
--recipe qwen3_8_27b_gguf \
--source gguf=Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.gguf \
--source vision=mmproj-model-bf16.gguf \
--components text,vision,mtp \
--resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
--proposal \
--name huihui-qwen3.8-27b-abliterated-underdog-saluki-swift15-iq2 \
--out models/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer
# file B adds:
# --source dflash2=/path/to/Qwen3.8-27B-DFlash2 --components text,vision,mtp,dflash2
Limitations
- Text tower at 2.351 bits/weight. No perplexity or KLD measurement is published.
- Abliterated model; refusal behaviour is intentionally altered.
- Video input is not supported by the engine.
- DFlash2 is trained against the base model and is reused here unchanged.
Credits and license
| Base model | Qwen/Qwen3.8-27B (Apache-2.0) |
| Abliteration | huihui-ai |
| GGUF (GSQ-RCO per-tensor block types) | huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF — Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.gguf. The file is its own authority on the method: it carries ISTA DASLab's GSQ-RCO metadata (general.name, general.description, general.quantized_by, quantize.imatrix.*) next to huihui-ai's abliteration |
| Vision tower | same repository — mmproj-model-bf16.gguf |
| DFlash2 drafter | z-lab/Qwen3.8-27B-DFlash2 |
| Converter and engine | iamwavecut/ninfer-all |
Apache-2.0, inherited from the base model.
简体中文
English | 简体中文
去审查版 Qwen3.8-27B 的 NInfer v3 产物(.ninfer)。文本塔、token 嵌入、输出头与 MTP 头就是
GSQ-RCO 逐张量混合精度 IQ2 级 GGUF(Underdog-Saluki Swift 1.5)的块,逐字节存入;视觉塔与 DFlash2
适配器则各自从其来源重新编码。每个文件自包含:权重、分词器、对话模板、生成配置、图像与视频预处理
配置、逻辑绑定。容器不是 GGUF 文件:transformers、llama.cpp 与 GGUF 运行时都读不了它,需由
NInfer v3 引擎加载。
谱系
| 环节 | 来源 |
|---|---|
| 去审查与 GSQ-RCO IQ2 量化 | huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF —— 本产物存储的文本塔、token 嵌入、输出头、MTP 头与视觉塔 |
| DFlash2 草稿头 | z-lab/Qwen3.8-27B-DFlash2 —— 仅 @@-dflash2@@ 文件 |
| 祖先基座 | Qwen/Qwen3.8-27B,Apache-2.0 |
| 本仓库 | 把这些权重重新编码进 NInfer v3 容器;此处不训练、也不重新量化任何权重 |
卡片元数据里的 base_model_relation: merge 记录的是同一条关系。
文件
| 文件 | 字节 | GiB | 组件 | SHA-256 |
|---|---|---|---|---|
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer |
8,850,451,200 | 8.24 | text, vision, mtp, proposal | c7295fd5ed1adc8371cb7290265cdc50037569c21df98acb101377bd0451a4f9 |
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer |
11,077,268,736 | 10.32 | text, vision, mtp, dflash2, proposal | 785e7b58cd3cdbfd07e7216ec5fb70647c238d623bd60697221eb9fd20a59df3 |
文件 B = 文件 A + DFlash2 草稿头(+2.13 GiB)。两者的文本塔、视觉塔、MTP 头与 proposal 头完全相同, 取其一即可。
下载
hf download NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer \
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer --local-dir models
hf download NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer \
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer --local-dir models
校验
sha256sum -c SHA256SUMS
仓库同时保存每个产物的 .conversion.json 报告,记录每个对象的来源、方法与格式(文件 A 1194 个:
1188 个张量 + 6 个资源;文件 B 1260 个)。
构成
| 组件 | 组装来源 | 表示形式 | 参数量 | 存储 |
|---|---|---|---|---|
| text:64 层、token 嵌入与输出头 | GSQ-RCO GGUF 自身的块 | gguf_blocks_v1 —— 每个张量保留量化器分配的 ggml 块类型(IQ1_S、IQ1_M、IQ2_XXS、IQ2_XS、IQ2_S、IQ3_XXS、IQ3_S、IQ4_XS、Q2_K、Q4_K、Q6_K、Q8_0),逐行逐字节拷贝;不做任何重新量化 |
26.896 B | 7.361 GiB |
| 视觉 | 同发布的 BF16 mmproj,重新编码 | q4_g64_fp16、q5_g64_fp16、q6_g64_fp16、q8_g32_fp16 |
0.461 B | 0.275 GiB |
| mtp | checkpoint 自带的 blk.64.nextn.* 张量 |
Q6_K 块 | 0.425 B | 0.324 GiB |
| proposal 头 | 输出头自身的行,收集而成 | IQ3_S,NInfer 排序中最高频的 131,072 个 token | 0.671 B | 0.269 GiB |
| dflash2(文件 B) | z-lab 的 Qwen3.8-27B-DFlash2,重新编码 | 投影为 q8_g32_fp16,参数为 bf16 |
1.977 B | 2.126 GiB |
| 合计,文件 A | 28.45 B | 8.24 GiB | ||
| 合计,文件 B | 30.43 B | 10.32 GiB |
GDN 控制量(ssm_alpha、ssm_beta)以 BF16 存储;范数、卷积、A_log、dt_bias 为精确值,
并还原导出器约定(w 而非 1 + w,A_log 由 -exp(A_log) 得到)。GDN 输出投影保留
llama.cpp 的分块 value head 顺序,以 input_columns Use 辅助量表达。
文件 A 有 7.905 GiB(96.1%)以 gguf_blocks_v1 存储;其余是 NInfer 视觉格式的视觉塔与 BF16/F32
参数张量。文件 B 另加 DFlash2 适配器,它同样不属于那个 GGUF。
模型
| 文本塔 | 64 层:48 层 GDN 线性注意力 + 16 层全注意力,每第四层为全注意力 |
| 注意力 | 24 个 query 头、4 个 KV 头、head_dim 256;输出门融进 query 投影;部分 RoPE 0.25,mRoPE 分段 [11, 11, 10],theta 1e7 |
| GDN | 16 个 key 头 × 128、48 个 value 头 × 128,卷积核 4,状态为 FP32 |
| 宽度 | hidden 5120、intermediate 17408、词表 248,320、原生窗口 262,144 token |
| 视觉塔 | 27 层、hidden 1152、intermediate 4304、16 头、patch 16、时间 patch 2、merge 2、2,304 位置 |
| 对话模板 | NInfer 固定的 qwen3_8.jinja:默认开启思考、effort xhigh、保留已结束轮次的推理 |
运行
需要 NInfer v3 引擎:iamwavecut/ninfer-all 的 master 分支(自行构建)或容器镜像。
内核由产物内的架构、配置与绑定决定,量化方式无需开关声明。
容器
docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
-v "$PWD/models:/models" -v ninfer-cache:/cache \
ghcr.io/iamwavecut/ninfer-all serve /models/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer \
--model-id huihui-qwen3.8-27b-iq2 --max-context 262144 --kv-capacity auto --kv-dtype int8 \
--gdn-state-fp16 --spec mtp --draft-tokens 4 --adaptive-mtp --lm-head-draft \
--vision --vision-residency overlay --vision-max-merged 1024
自建版本
文件 A 用 MTP 头起草:
ninfer-serve Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer \
--model-id huihui-qwen3.8-27b-iq2 --host 127.0.0.1 --port 1145 \
--max-context 262144 --kv-capacity auto --kv-dtype int8 --gdn-state-fp16 \
--spec mtp --draft-tokens 4 --adaptive-mtp --lm-head-draft \
--vision --vision-residency resident --vision-max-merged 1024 \
--prefill-cublas
文件 B 用 DFlash2 适配器起草:
ninfer-serve Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer \
--model-id huihui-qwen3.8-27b-iq2-dflash2 --host 127.0.0.1 --port 1145 \
--max-context 32768 --kv-capacity auto --kv-dtype rk8v4 --gdn-state-fp16 \
--spec dflash2 --draft-tokens 5 --lm-head-draft \
--vision --vision-residency overlay --vision-max-merged 1024
服务在 /v1 上应答 OpenAI、Anthropic 与 Responses 请求;自建版本默认监听 127.0.0.1:8080,
--host 与 --port 可改。
参数
| 参数 | 作用 |
|---|---|
--kv-dtype |
KV 存储:bf16、int8、fp8、nvfp4、k8v4、rk8v4、rk4v4、rk4v4-e8、rk2v4-e8 |
--kv-capacity N|auto |
共享 KV 容量;auto 取权重之外的空闲显存 |
--vision-residency |
resident(设备)、overlay(每张图借用设备显存)、cpu |
--vision-max-merged |
单张图片的合并 token 数,64..16384 |
--lm-head-draft |
改用 proposal 头起草并加载它,而非完整输出头 |
--gdn-state-fp16 |
GDN 循环状态保持 FP16 |
--spec、--draft-tokens |
投机解码后端与每轮草稿数,1..15 |
显存
文件 A 载入 8.24 GiB 权重,文件 B 载入 10.32 GiB。两者都在 16 GB 笔记本 GPU 上以
--kv-capacity auto 运行过;同一张卡上文件 B 留下的 KV 缓存更小。视觉塔另占 0.275 GiB 权重
(常驻或按图借用),且不支持视频。
引擎兼容性
| 能力 | 要求 |
|---|---|
| 文件 A —— MTP 起草 | 任何支持 --spec mtp 的 NInfer v3 构建 |
| 文件 B —— DFlash2 起草 | 接受 --spec dflash2 的构建;仅支持 MTP 的引擎会在启动时拒绝该参数,此时改用文件 A |
| 视觉 | 文本与图片,两个文件均可 |
| proposal 头 | --lm-head-draft |
Windows 下运行时路径、模型路径与模型名须为 ASCII。
转换
官方 recipe qwen3_8_27b_gguf,直接沿用模型自己的块:
python3 -m tools.convert \
--model /path/to/Qwen3.8-27B \ # 仅需 config.json 与分词器文件
--recipe qwen3_8_27b_gguf \
--source gguf=Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.gguf \
--source vision=mmproj-model-bf16.gguf \
--components text,vision,mtp \
--resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
--proposal \
--name huihui-qwen3.8-27b-abliterated-underdog-saluki-swift15-iq2 \
--out models/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer
# 文件 B 追加:
# --source dflash2=/path/to/Qwen3.8-27B-DFlash2 --components text,vision,mtp,dflash2
限制
- 文本塔 2.351 位/权重。未发布困惑度或 KLD 测量。
- 去审查模型,拒答行为被有意改变。
- 引擎不支持视频输入。
- DFlash2 针对基座模型训练,此处原样复用。
出处与许可
| 基座模型 | Qwen/Qwen3.8-27B(Apache-2.0) |
| 去审查 | huihui-ai |
| GGUF(GSQ-RCO 逐张量块类型) | huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF —— Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.gguf。方法归属以该文件自身为准:头部同时带 ISTA DASLab 的 GSQ-RCO 元数据(general.name、general.description、general.quantized_by、quantize.imatrix.*)与 huihui-ai 的去审查标记 |
| 视觉塔 | 同一仓库 —— mmproj-model-bf16.gguf |
| DFlash2 草稿头 | z-lab/Qwen3.8-27B-DFlash2 |
| 转换器与引擎 | iamwavecut/ninfer-all |
Apache-2.0,继承自基座模型。
- Downloads last month
- 3,144