Instructions to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF # Run inference directly in the terminal: llama cli -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF # Run inference directly in the terminal: llama cli -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF # Run inference directly in the terminal: ./build/bin/llama cli -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Use Docker
docker model run hf.co/andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
- Llama new
- LM Studio
- Jan
- vLLM
How to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
- Ollama
How to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with Ollama:
ollama run hf.co/andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
- Unsloth Desktop
- Pi
How to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with Docker Model Runner:
docker model run hf.co/andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
- Lemonade
How to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Run and chat with the model
lemonade run user.Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Huihui-Nex-N2-mini-abliterated โ APEX I-Quality + MTP (GGUF)
GGUF quantizations of huihui-ai/Huihui-Nex-N2-mini-abliterated using the APEX (Adaptive Precision for EXpert Models) MoE-aware mixed-precision technique, with the built-in MTP (Multi-Token-Prediction) draft head retained for speculative decoding.
- Base arch:
qwen3_5_moe(Qwen3.5-style hybrid MoE: 40 layers, 256 experts / 8 active, linear-attention SSM + full-attention, MRoPE, multimodal vision projector) - Parameters: ~35.1B total (A3B active)
- MTP: built-in 1-layer nextn draft head (kept embedded in the GGUF โ no separate draft model needed)
Files
Both MTP (built-in Multi-Token-Prediction draft head retained for speculative decoding) and non-MTP (clean, max compatibility) variants are provided:
| File | Tier | MTP | Size | BPW | imatrix | Notes |
|---|---|---|---|---|---|---|
Huihui-Nex-N2-mini-APEX-I-Quality-MTP.gguf |
APEX I-Quality | โ | ~24.5 GB | 5.52 | โ diverse | Best accuracy + speculative decoding (needs recent llama.cpp w/ bundled-MTP support) |
Huihui-Nex-N2-mini-APEX-Quality-MTP.gguf |
APEX Quality | โ | ~24.5 GB | 5.52 | โ | Lowest perplexity + speculative decoding |
Huihui-Nex-N2-mini-APEX-I-Quality-noMTP.gguf |
APEX I-Quality | โ | ~22.8 GB | 5.52 | โ diverse | Best accuracy, loads on any llama.cpp build |
Huihui-Nex-N2-mini-APEX-Quality-noMTP.gguf |
APEX Quality | โ | ~22.8 GB | 5.52 | โ | Lowest perplexity, max compatibility |
Which to pick?
- MTP variants โ faster tok/s via speculative decoding, but need a recent
llama.cpp(bundled-MTP loader support, PR #22673 or newer;c1304d7confirmed working). - non-MTP variants โ load on any
llama.cppbuild (incl. older), slightly smaller. Drop-in if your tool reportsmissing tensor 'blk.40.ssm_conv1d.weight'on the MTP file.
The MTP variants include the full draft block as blk.40.* + blk.40.nextn.{eh_proj,enorm,hnorm,shared_head_norm} tensors, with the MTP projection (eh_proj) kept at F16 and norms at F32 (per the APEX-MTP methodology: the draft head must stay high-precision or speculative-decoding acceptance drops). Non-MTP variants have block_count=40 (no blk.40.* / no nextn_predict_layers metadata).
APEX tier details (Quality profile)
APEX assigns precision per tensor role + per layer:
- Edge layers (L0-4, L35-39): routed experts
Q6_K - Near-edge (L5-9, L30-34): routed experts
Q5_K - Middle (L10-29): routed experts
IQ4_XS - Shared expert:
Q8_0(heavy-tailed, kurtosis ~13 โ needs high precision) - Attention:
Q6_K - MTP draft head (blk.40.*):
F16(override)
I-variant (imatrix)
APEX-I-Quality is calibrated with a diverse importance matrix (chat + code + reasoning + tool-calling, no Wikipedia) rather than encyclopedic text. This trades a tiny wikitext perplexity increase for better real-world (assistant/coding/tool-use) accuracy and lower KL divergence. See the APEX technical report.
Note: the imatrix here is a locally-built diverse calibration set (~560 documents sampled from UltraChat, Alpaca, GSM8K chain-of-thought, Hermes function-calling, HumanEval code), not the private APEX
calibration_v1.2.txt. Methodology matches; results are not byte-identical to upstream APEX I-variants.
Loading / Inference
- Built-in MTP is load-bearing: use a recent
llama.cppbuild that supports bundled MTP / nextn (llama + spec: MTP SupportPR #22673 or newer โc1304d7confirmed working). Older builds fail withmissing tensor 'blk.40.ssm_conv1d.weight'. - MTP speculative decoding activates automatically on load โ faster tok/s, no
-mddraft flag needed.
# MTP variant (needs recent llama.cpp with bundled-MTP support)
llama-server -m Huihui-Nex-N2-mini-APEX-I-Quality-MTP.gguf -ngl 12 -c 8192 --port 8080
# non-MTP variant (any llama.cpp build)
llama-server -m Huihui-Nex-N2-mini-APEX-I-Quality-noMTP.gguf -ngl 12 -c 8192 --port 8080
On a 12 GB VRAM GPU (e.g. RTX 3060), -ngl around 8-12 fits the ~23 GB model with partial CPU offload.
โ ๏ธ Usage warnings (abliterated / uncensored)
This model is an abliterated (uncensored) derivative โ its refusal direction has been removed via direction-ablation. The original huihui-ai warnings apply:
- Risk of sensitive/controversial outputs โ safety filtering is significantly reduced.
- Not suitable for all audiences โ outputs may be inappropriate for public settings, underage users, or high-security applications.
- Legal & ethical responsibility โ ensure your usage complies with local laws. You are solely responsible for any consequences.
- Research / experimental use recommended โ avoid unmonitored production or public-facing deployment.
- No default safety guarantees โ this model has not undergone rigorous safety optimization. The uploader bears no responsibility for any consequences arising from its use.
Credits
- Original model:
nex-agi/Nex-N2-mini - Abliteration:
huihui-ai - Quantization method: APEX โ Adaptive Precision for EXpert Models by the LocalAI team (Ettore Di Giacinto & Richard Palethorpe)
- Engine: llama.cpp
Donation
If this is useful, donations are appreciated:
BTC: bc1q6xxf0j3e7zn52cqrprc6gplql225wj8mnq75yw
- Downloads last month
- 420
We're not able to determine the quantization variants.
Model tree for andrevp/Huihui-Nex-N2-mini-abliterated-APEX-MTP-GGUF
Base model
huihui-ai/Huihui-Nex-N2-mini-abliterated