Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2

English | 简体中文

NInfer v3 artifacts (.ninfer) of the abliterated Qwen3.8-27B. The text tower, token embedding, output head and MTP head are the blocks of the GSQ-RCO mixed-precision IQ2-class GGUF (Underdog-Saluki Swift 1.5), stored byte for byte; the Vision tower and the DFlash2 adapter are re-encoded from their own sources. Each file is self-contained: weights, tokenizer, chat template, generation config, image and video preprocessor configs, logical bindings. The container is not a GGUF file: transformers, llama.cpp and GGUF runtimes cannot read it, an NInfer v3 engine loads it.

Lineage

Step Source
Abliteration and GSQ-RCO IQ2 quantization huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF — the text tower, token embedding, output head, MTP head and Vision tower these files store
DFlash2 drafter z-lab/Qwen3.8-27B-DFlash2 — the @@-dflash2@@ file only
Ancestral base Qwen/Qwen3.8-27B, Apache-2.0
This repository re-encodes those weights into NInfer v3 containers; nothing is retrained or re-quantized here

base_model_relation: merge records the same relation in the card metadata.

Files

File Bytes GiB Components SHA-256
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer 8,850,451,200 8.24 text, vision, mtp, proposal c7295fd5ed1adc8371cb7290265cdc50037569c21df98acb101377bd0451a4f9
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer 11,077,268,736 10.32 text, vision, mtp, dflash2, proposal 785e7b58cd3cdbfd07e7216ec5fb70647c238d623bd60697221eb9fd20a59df3

File B is file A plus the DFlash2 drafter (+2.13 GiB). Both carry the same text tower, Vision tower, MTP head and proposal head, so download one.

Download

hf download NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer \
  Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer --local-dir models

hf download NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer \
  Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer --local-dir models

Verify

sha256sum -c SHA256SUMS

The repository also carries each artifact's .conversion.json report, recording the source, method and format of every object (1194 objects in file A: 1188 tensors and 6 resources; 1260 in file B).

What is inside

Component Assembled from Representation Parameters Stored
text: the 64 layers, token embedding and output head the GSQ-RCO GGUF's own blocks gguf_blocks_v1 — each tensor keeps the ggml block type the quantizer assigned it (IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_XS, Q2_K, Q4_K, Q6_K, Q8_0) and is copied row for row, byte for byte; nothing is re-quantized 26.896 B 7.361 GiB
vision the release's BF16 mmproj, re-encoded q4_g64_fp16, q5_g64_fp16, q6_g64_fp16, q8_g32_fp16 0.461 B 0.275 GiB
mtp the checkpoint's own blk.64.nextn.* tensors Q6_K blocks 0.425 B 0.324 GiB
proposal head the output head's own rows, gathered IQ3_S, the 131,072 most frequent tokens of NInfer's ranking 0.671 B 0.269 GiB
dflash2 (file B) z-lab's Qwen3.8-27B-DFlash2, re-encoded q8_g32_fp16 projections, bf16 parameters 1.977 B 2.126 GiB
total, file A 28.45 B 8.24 GiB
total, file B 30.43 B 10.32 GiB

GDN controls (ssm_alpha, ssm_beta) are stored as BF16; norms, convolution, A_log and dt_bias are exact, with the exporter's conventions restored (w instead of 1 + w, A_log from -exp(A_log)). The GDN output projection keeps llama.cpp's tiled value-head order as an input_columns Use auxiliary.

7.905 GiB of file A (96.1%) is stored in gguf_blocks_v1; the rest is the Vision tower in NInfer's Vision formats and the BF16/F32 parameter tensors. File B adds the DFlash2 adapter, which is not part of the GGUF either.

The model

Text tower 64 layers: 48 GDN linear-attention and 16 full-attention layers, every fourth
Attention 24 query heads, 4 KV heads, head_dim 256; the output gate is fused into the query projection; partial RoPE 0.25, mRoPE sections [11, 11, 10], theta 1e7
GDN 16 key heads × 128, 48 value heads × 128, convolution kernel 4, state in FP32
Widths hidden 5120, intermediate 17408, vocabulary 248,320, native window 262,144 tokens
Vision tower 27 blocks, hidden 1152, intermediate 4304, 16 heads, patch 16, temporal patch 2, merge 2, 2,304 positions
Chat template NInfer's pinned qwen3_8.jinja: thinking on by default, effort xhigh, closed-turn reasoning retained

Running

An NInfer v3 engine is required: the master branch of iamwavecut/ninfer-all, built, or the container image. Kernels are selected from the architecture, configuration and bindings stored in the artifact, so no flag declares the quantization.

Container

docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer \
  --model-id huihui-qwen3.8-27b-iq2 --max-context 262144 --kv-capacity auto --kv-dtype int8 \
  --gdn-state-fp16 --spec mtp --draft-tokens 4 --adaptive-mtp --lm-head-draft \
  --vision --vision-residency overlay --vision-max-merged 1024

A build

File A drafts with the MTP head:

ninfer-serve Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer \
    --model-id huihui-qwen3.8-27b-iq2 --host 127.0.0.1 --port 1145 \
    --max-context 262144 --kv-capacity auto --kv-dtype int8 --gdn-state-fp16 \
    --spec mtp --draft-tokens 4 --adaptive-mtp --lm-head-draft \
    --vision --vision-residency resident --vision-max-merged 1024 \
    --prefill-cublas

File B drafts with the DFlash2 adapter:

ninfer-serve Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer \
    --model-id huihui-qwen3.8-27b-iq2-dflash2 --host 127.0.0.1 --port 1145 \
    --max-context 32768 --kv-capacity auto --kv-dtype rk8v4 --gdn-state-fp16 \
    --spec dflash2 --draft-tokens 5 --lm-head-draft \
    --vision --vision-residency overlay --vision-max-merged 1024

The server answers OpenAI, Anthropic and Responses requests under /v1; a build listens on 127.0.0.1:8080 unless --host and --port say otherwise.

Flags

Flag Effect
--kv-dtype KV storage: bf16, int8, fp8, nvfp4, k8v4, rk8v4, rk4v4, rk4v4-e8, rk2v4-e8
--kv-capacity N|auto shared KV capacity; auto takes what is free after the weights
--vision-residency resident (device), overlay (device memory borrowed per image), cpu
--vision-max-merged merged tokens of one image, 64..16384
--lm-head-draft draft through the proposal head instead of the full output head, and load it
--gdn-state-fp16 keep the GDN recurrent state in FP16
--spec, --draft-tokens speculative backend and drafts per round, 1..15

Memory

File A loads 8.24 GiB of weights and file B 10.32 GiB. Both ran on a 16 GB laptop GPU with --kv-capacity auto; file B leaves the smaller KV cache on the same card. The Vision tower adds 0.275 GiB of weights, resident or borrowed, and video is not supported.

Engine compatibility

Capability Requirement
File A — MTP drafting any NInfer v3 build with --spec mtp
File B — DFlash2 drafting a build that accepts --spec dflash2; an engine limited to MTP refuses the flag at startup and must run file A
Vision text and image, in both files
Proposal head --lm-head-draft

On Windows the runtime path, model path and model name must be ASCII.

Conversion

Official recipe qwen3_8_27b_gguf, from the model's own blocks:

python3 -m tools.convert \
  --model /path/to/Qwen3.8-27B \                       # config.json + tokenizer files only
  --recipe qwen3_8_27b_gguf \
  --source gguf=Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.gguf \
  --source vision=mmproj-model-bf16.gguf \
  --components text,vision,mtp \
  --resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
  --proposal \
  --name huihui-qwen3.8-27b-abliterated-underdog-saluki-swift15-iq2 \
  --out models/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer

# file B adds:
#   --source dflash2=/path/to/Qwen3.8-27B-DFlash2 --components text,vision,mtp,dflash2

Limitations

  • Text tower at 2.351 bits/weight. No perplexity or KLD measurement is published.
  • Abliterated model; refusal behaviour is intentionally altered.
  • Video input is not supported by the engine.
  • DFlash2 is trained against the base model and is reused here unchanged.

Credits and license

Base model Qwen/Qwen3.8-27B (Apache-2.0)
Abliteration huihui-ai
GGUF (GSQ-RCO per-tensor block types) huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF — Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.gguf. The file is its own authority on the method: it carries ISTA DASLab's GSQ-RCO metadata (general.name, general.description, general.quantized_by, quantize.imatrix.*) next to huihui-ai's abliteration
Vision tower same repository — mmproj-model-bf16.gguf
DFlash2 drafter z-lab/Qwen3.8-27B-DFlash2
Converter and engine iamwavecut/ninfer-all

Apache-2.0, inherited from the base model.


简体中文

English | 简体中文

去审查版 Qwen3.8-27B 的 NInfer v3 产物(.ninfer)。文本塔、token 嵌入、输出头与 MTP 头就是 GSQ-RCO 逐张量混合精度 IQ2 级 GGUF(Underdog-Saluki Swift 1.5)的块,逐字节存入;视觉塔与 DFlash2 适配器则各自从其来源重新编码。每个文件自包含:权重、分词器、对话模板、生成配置、图像与视频预处理 配置、逻辑绑定。容器不是 GGUF 文件:transformers、llama.cpp 与 GGUF 运行时都读不了它,需由 NInfer v3 引擎加载。

谱系

环节 来源
去审查与 GSQ-RCO IQ2 量化 huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF —— 本产物存储的文本塔、token 嵌入、输出头、MTP 头与视觉塔
DFlash2 草稿头 z-lab/Qwen3.8-27B-DFlash2 —— 仅 @@-dflash2@@ 文件
祖先基座 Qwen/Qwen3.8-27B,Apache-2.0
本仓库 把这些权重重新编码进 NInfer v3 容器;此处不训练、也不重新量化任何权重

卡片元数据里的 base_model_relation: merge 记录的是同一条关系。

文件

文件 字节 GiB 组件 SHA-256
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer 8,850,451,200 8.24 text, vision, mtp, proposal c7295fd5ed1adc8371cb7290265cdc50037569c21df98acb101377bd0451a4f9
Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer 11,077,268,736 10.32 text, vision, mtp, dflash2, proposal 785e7b58cd3cdbfd07e7216ec5fb70647c238d623bd60697221eb9fd20a59df3

文件 B = 文件 A + DFlash2 草稿头(+2.13 GiB)。两者的文本塔、视觉塔、MTP 头与 proposal 头完全相同, 取其一即可。

下载

hf download NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer \
  Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer --local-dir models

hf download NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer \
  Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer --local-dir models

校验

sha256sum -c SHA256SUMS

仓库同时保存每个产物的 .conversion.json 报告,记录每个对象的来源、方法与格式(文件 A 1194 个: 1188 个张量 + 6 个资源;文件 B 1260 个)。

构成

组件 组装来源 表示形式 参数量 存储
text:64 层、token 嵌入与输出头 GSQ-RCO GGUF 自身的块 gguf_blocks_v1 —— 每个张量保留量化器分配的 ggml 块类型(IQ1_S、IQ1_M、IQ2_XXS、IQ2_XS、IQ2_S、IQ3_XXS、IQ3_S、IQ4_XS、Q2_K、Q4_K、Q6_K、Q8_0),逐行逐字节拷贝;不做任何重新量化 26.896 B 7.361 GiB
视觉 同发布的 BF16 mmproj,重新编码 q4_g64_fp16、q5_g64_fp16、q6_g64_fp16、q8_g32_fp16 0.461 B 0.275 GiB
mtp checkpoint 自带的 blk.64.nextn.* 张量 Q6_K 块 0.425 B 0.324 GiB
proposal 头 输出头自身的行,收集而成 IQ3_S,NInfer 排序中最高频的 131,072 个 token 0.671 B 0.269 GiB
dflash2(文件 B) z-lab 的 Qwen3.8-27B-DFlash2,重新编码 投影为 q8_g32_fp16,参数为 bf16 1.977 B 2.126 GiB
合计,文件 A 28.45 B 8.24 GiB
合计,文件 B 30.43 B 10.32 GiB

GDN 控制量(ssm_alpha、ssm_beta)以 BF16 存储;范数、卷积、A_log、dt_bias 为精确值, 并还原导出器约定(w 而非 1 + w,A_log 由 -exp(A_log) 得到)。GDN 输出投影保留 llama.cpp 的分块 value head 顺序,以 input_columns Use 辅助量表达。

文件 A 有 7.905 GiB(96.1%)以 gguf_blocks_v1 存储;其余是 NInfer 视觉格式的视觉塔与 BF16/F32 参数张量。文件 B 另加 DFlash2 适配器,它同样不属于那个 GGUF。

模型

文本塔 64 层:48 层 GDN 线性注意力 + 16 层全注意力,每第四层为全注意力
注意力 24 个 query 头、4 个 KV 头、head_dim 256;输出门融进 query 投影;部分 RoPE 0.25,mRoPE 分段 [11, 11, 10],theta 1e7
GDN 16 个 key 头 × 128、48 个 value 头 × 128,卷积核 4,状态为 FP32
宽度 hidden 5120、intermediate 17408、词表 248,320、原生窗口 262,144 token
视觉塔 27 层、hidden 1152、intermediate 4304、16 头、patch 16、时间 patch 2、merge 2、2,304 位置
对话模板 NInfer 固定的 qwen3_8.jinja:默认开启思考、effort xhigh、保留已结束轮次的推理

运行

需要 NInfer v3 引擎:iamwavecut/ninfer-all 的 master 分支(自行构建)或容器镜像。 内核由产物内的架构、配置与绑定决定,量化方式无需开关声明。

容器

docker run --rm --gpus all -p 8080:8080 --ulimit memlock=-1 \
  -v "$PWD/models:/models" -v ninfer-cache:/cache \
  ghcr.io/iamwavecut/ninfer-all serve /models/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer \
  --model-id huihui-qwen3.8-27b-iq2 --max-context 262144 --kv-capacity auto --kv-dtype int8 \
  --gdn-state-fp16 --spec mtp --draft-tokens 4 --adaptive-mtp --lm-head-draft \
  --vision --vision-residency overlay --vision-max-merged 1024

自建版本

文件 A 用 MTP 头起草:

ninfer-serve Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer \
    --model-id huihui-qwen3.8-27b-iq2 --host 127.0.0.1 --port 1145 \
    --max-context 262144 --kv-capacity auto --kv-dtype int8 --gdn-state-fp16 \
    --spec mtp --draft-tokens 4 --adaptive-mtp --lm-head-draft \
    --vision --vision-residency resident --vision-max-merged 1024 \
    --prefill-cublas

文件 B 用 DFlash2 适配器起草:

ninfer-serve Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp-dflash2.ninfer \
    --model-id huihui-qwen3.8-27b-iq2-dflash2 --host 127.0.0.1 --port 1145 \
    --max-context 32768 --kv-capacity auto --kv-dtype rk8v4 --gdn-state-fp16 \
    --spec dflash2 --draft-tokens 5 --lm-head-draft \
    --vision --vision-residency overlay --vision-max-merged 1024

服务在 /v1 上应答 OpenAI、Anthropic 与 Responses 请求;自建版本默认监听 127.0.0.1:8080, --host 与 --port 可改。

参数

参数 作用
--kv-dtype KV 存储:bf16、int8、fp8、nvfp4、k8v4、rk8v4、rk4v4、rk4v4-e8、rk2v4-e8
--kv-capacity N|auto 共享 KV 容量;auto 取权重之外的空闲显存
--vision-residency resident(设备)、overlay(每张图借用设备显存)、cpu
--vision-max-merged 单张图片的合并 token 数,64..16384
--lm-head-draft 改用 proposal 头起草并加载它,而非完整输出头
--gdn-state-fp16 GDN 循环状态保持 FP16
--spec、--draft-tokens 投机解码后端与每轮草稿数,1..15

显存

文件 A 载入 8.24 GiB 权重,文件 B 载入 10.32 GiB。两者都在 16 GB 笔记本 GPU 上以 --kv-capacity auto 运行过;同一张卡上文件 B 留下的 KV 缓存更小。视觉塔另占 0.275 GiB 权重 (常驻或按图借用),且不支持视频。

引擎兼容性

能力 要求
文件 A —— MTP 起草 任何支持 --spec mtp 的 NInfer v3 构建
文件 B —— DFlash2 起草 接受 --spec dflash2 的构建;仅支持 MTP 的引擎会在启动时拒绝该参数,此时改用文件 A
视觉 文本与图片,两个文件均可
proposal 头 --lm-head-draft

Windows 下运行时路径、模型路径与模型名须为 ASCII。

转换

官方 recipe qwen3_8_27b_gguf,直接沿用模型自己的块:

python3 -m tools.convert \
  --model /path/to/Qwen3.8-27B \                       # 仅需 config.json 与分词器文件
  --recipe qwen3_8_27b_gguf \
  --source gguf=Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.gguf \
  --source vision=mmproj-model-bf16.gguf \
  --components text,vision,mtp \
  --resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
  --proposal \
  --name huihui-qwen3.8-27b-abliterated-underdog-saluki-swift15-iq2 \
  --out models/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.ninfer

# 文件 B 追加:
#   --source dflash2=/path/to/Qwen3.8-27B-DFlash2 --components text,vision,mtp,dflash2

限制

  • 文本塔 2.351 位/权重。未发布困惑度或 KLD 测量。
  • 去审查模型,拒答行为被有意改变。
  • 引擎不支持视频输入。
  • DFlash2 针对基座模型训练,此处原样复用。

出处与许可

基座模型 Qwen/Qwen3.8-27B(Apache-2.0)
去审查 huihui-ai
GGUF(GSQ-RCO 逐张量块类型) huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF —— Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-mtp.gguf。方法归属以该文件自身为准:头部同时带 ISTA DASLab 的 GSQ-RCO 元数据(general.name、general.description、general.quantized_by、quantize.imatrix.*)与 huihui-ai 的去审查标记
视觉塔 同一仓库 —— mmproj-model-bf16.gguf
DFlash2 草稿头 z-lab/Qwen3.8-27B-DFlash2
转换器与引擎 iamwavecut/ninfer-all

Apache-2.0,继承自基座模型。

Downloads last month
3,144
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for NateVoid/Huihui-Qwen3.8-27B-abliterated-Underdog-Saluki-Swift-1.5-IQ2-Ninfer