Instructions to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL # Run inference directly in the terminal: ./build/bin/llama cli -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
Use Docker
docker model run hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
- Llama new
- LM Studio
- Jan
- vLLM
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
- SGLang
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Ollama
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with Ollama:
ollama run hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
- Unsloth Desktop
- Pi
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with Docker Model Runner:
docker model run hf.co/huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
- Lemonade
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
Run and chat with the model
lemonade run user.Huihui-Qwen3.8-27B-abliterated-GGUF-UD-Q4_K_XL
List all available models
lemonade list
- Hermes Agent
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "huihui-ai/Huihui-Qwen3.8-27B-abliterated-GGUF:UD-Q4_K_XL" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
The newly added Huihui-Qwen3.8-27B-abliterated-GSQ-RCO is awesome
@huihui-ai Hi just want to say thank you for the newly added Huihui-Qwen3.8-27B-abliterated-GSQ-RCO is amazing and seems to be even faster than the Unsloth Quant
my hardware is not the best only 8GB Vram and 4070
anyone know when GSQ-RCO iq4_xs comes out?
(or gsq-rco 14.4 gigs)
same for dw-ud iq4_xs, i really wanna test kld between both since theyre currently considered the front runners for quantization.
Edit:
Firstly thank you to the fellas who showed me the full pic about gsq-rco over 3.7bpw.
Now as for the ud-dw iq4_xs matter - I still don't understand the reason y'all never produced it, I mean ud-iq4_xs is pretty good and I can run it while only consuming 15.5 gigs of vram on an optimized system, that leaves 500MB avaliable which should be more then enough for the ud-dw variant since dw variants usually add about 300MB to file size.
The difference in performance is just to big to not feel like I'm missing out.
Note:
I know many pepole say leave 1 gig of room avaliable and I've got a few things to say about that:
Firstly unless your runing agentic workloads requiring massive eval and physical batches you don't need more then 500MB of free vram
Secondly I usually do leave the extra 500MB of space for eval and physical batches but ud-dw iq4_xs benchmarks have shown to genuinely rival q6_k and trutly optimized systems will still have around 300MB left for eval and physical batches which should be just about enough.
To conclude:
GIVE US UD-DW IQ4_XS PLEAS OH HUIHUI LORDS FROM ABOVE.
/
|
This part was satire i haven't gone insane fellas dw.
@huihui-ai Not sure if you already saw this, but I thought it was interesting.
A group is claiming a free inference harness (ledger-based self-orchestration) lifts Qwen3.8-27B on LiveCodeBench Hard enough to match or slightly beat Claude Fable 5βs single-call score. They released the harness and the paper.
Reddit: https://www.reddit.com/r/Qwen_AI/s/jocmCf3ZLr
Paper: https://arxiv.org/abs/2608.26480
Code: https://github.com/slee-persis/GVS5H
Those of you who have these accounts can do a comparison test on huihui-ai/Huihui-Qwen3.8-27B-abliterated.
anyone know when GSQ-RCO iq4_xs comes out?
(or gsq-rco 14.4 gigs)
same for dw-ud iq4_xs, i really wanna test kld between both since theyre currently considered the front runners for quantization.
When the model quantization level is above 3.7 bpw, the GSQ RCO algorithm no longer provides additional gains.
ok I tried it, and it failed to undertake any tool calls. It is good for chat, and does not code at all.
