Huihui-Nex-N2-mini-abliterated โ€” APEX I-Quality + MTP (GGUF)

GGUF quantizations of huihui-ai/Huihui-Nex-N2-mini-abliterated using the APEX (Adaptive Precision for EXpert Models) MoE-aware mixed-precision technique, with the built-in MTP (Multi-Token-Prediction) draft head retained for speculative decoding.

  • Base arch: qwen3_5_moe (Qwen3.5-style hybrid MoE: 40 layers, 256 experts / 8 active, linear-attention SSM + full-attention, MRoPE, multimodal vision projector)
  • Parameters: ~35.1B total (A3B active)
  • MTP: built-in 1-layer nextn draft head (kept embedded in the GGUF โ†’ no separate draft model needed)

Files

Both MTP (built-in Multi-Token-Prediction draft head retained for speculative decoding) and non-MTP (clean, max compatibility) variants are provided:

File Tier MTP Size BPW imatrix Notes
Huihui-Nex-N2-mini-APEX-I-Quality-MTP.gguf APEX I-Quality โœ… ~24.5 GB 5.52 โœ… diverse Best accuracy + speculative decoding (needs recent llama.cpp w/ bundled-MTP support)
Huihui-Nex-N2-mini-APEX-Quality-MTP.gguf APEX Quality โœ… ~24.5 GB 5.52 โŒ Lowest perplexity + speculative decoding
Huihui-Nex-N2-mini-APEX-I-Quality-noMTP.gguf APEX I-Quality โŒ ~22.8 GB 5.52 โœ… diverse Best accuracy, loads on any llama.cpp build
Huihui-Nex-N2-mini-APEX-Quality-noMTP.gguf APEX Quality โŒ ~22.8 GB 5.52 โŒ Lowest perplexity, max compatibility

Which to pick?

  • MTP variants โ†’ faster tok/s via speculative decoding, but need a recent llama.cpp (bundled-MTP loader support, PR #22673 or newer; c1304d7 confirmed working).
  • non-MTP variants โ†’ load on any llama.cpp build (incl. older), slightly smaller. Drop-in if your tool reports missing tensor 'blk.40.ssm_conv1d.weight' on the MTP file.

The MTP variants include the full draft block as blk.40.* + blk.40.nextn.{eh_proj,enorm,hnorm,shared_head_norm} tensors, with the MTP projection (eh_proj) kept at F16 and norms at F32 (per the APEX-MTP methodology: the draft head must stay high-precision or speculative-decoding acceptance drops). Non-MTP variants have block_count=40 (no blk.40.* / no nextn_predict_layers metadata).

APEX tier details (Quality profile)

APEX assigns precision per tensor role + per layer:

  • Edge layers (L0-4, L35-39): routed experts Q6_K
  • Near-edge (L5-9, L30-34): routed experts Q5_K
  • Middle (L10-29): routed experts IQ4_XS
  • Shared expert: Q8_0 (heavy-tailed, kurtosis ~13 โ†’ needs high precision)
  • Attention: Q6_K
  • MTP draft head (blk.40.*): F16 (override)

I-variant (imatrix)

APEX-I-Quality is calibrated with a diverse importance matrix (chat + code + reasoning + tool-calling, no Wikipedia) rather than encyclopedic text. This trades a tiny wikitext perplexity increase for better real-world (assistant/coding/tool-use) accuracy and lower KL divergence. See the APEX technical report.

Note: the imatrix here is a locally-built diverse calibration set (~560 documents sampled from UltraChat, Alpaca, GSM8K chain-of-thought, Hermes function-calling, HumanEval code), not the private APEX calibration_v1.2.txt. Methodology matches; results are not byte-identical to upstream APEX I-variants.

Loading / Inference

  • Built-in MTP is load-bearing: use a recent llama.cpp build that supports bundled MTP / nextn (llama + spec: MTP Support PR #22673 or newer โ€” c1304d7 confirmed working). Older builds fail with missing tensor 'blk.40.ssm_conv1d.weight'.
  • MTP speculative decoding activates automatically on load โ†’ faster tok/s, no -md draft flag needed.
# MTP variant (needs recent llama.cpp with bundled-MTP support)
llama-server -m Huihui-Nex-N2-mini-APEX-I-Quality-MTP.gguf -ngl 12 -c 8192 --port 8080

# non-MTP variant (any llama.cpp build)
llama-server -m Huihui-Nex-N2-mini-APEX-I-Quality-noMTP.gguf -ngl 12 -c 8192 --port 8080

On a 12 GB VRAM GPU (e.g. RTX 3060), -ngl around 8-12 fits the ~23 GB model with partial CPU offload.

โš ๏ธ Usage warnings (abliterated / uncensored)

This model is an abliterated (uncensored) derivative โ€” its refusal direction has been removed via direction-ablation. The original huihui-ai warnings apply:

  • Risk of sensitive/controversial outputs โ€” safety filtering is significantly reduced.
  • Not suitable for all audiences โ€” outputs may be inappropriate for public settings, underage users, or high-security applications.
  • Legal & ethical responsibility โ€” ensure your usage complies with local laws. You are solely responsible for any consequences.
  • Research / experimental use recommended โ€” avoid unmonitored production or public-facing deployment.
  • No default safety guarantees โ€” this model has not undergone rigorous safety optimization. The uploader bears no responsibility for any consequences arising from its use.

Credits

Donation

If this is useful, donations are appreciated:

BTC: bc1q6xxf0j3e7zn52cqrprc6gplql225wj8mnq75yw

Downloads last month
420
GGUF
Model size
35B params
Architecture
qwen35moe
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF

Quantized
(7)
this model