Instructions to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0 # Run inference directly in the terminal: llama cli -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0 # Run inference directly in the terminal: llama cli -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
Use Docker
docker model run hf.co/oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
- Ollama
How to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with Ollama:
ollama run hf.co/oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
- Unsloth Desktop
- Pi
How to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with Docker Model Runner:
docker model run hf.co/oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
- Lemonade
How to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
Run and chat with the model
lemonade run user.Sanshoku-Qwen3.8-3.6-27B-blend-GGUF-Q2_0
List all available models
lemonade list
- Hermes Agent
How to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF:Q2_0" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
๐ก Sanshoku-Qwen3.8-3.6-27B-blend-GGUF
A ternary model distilled from a merge of Qwen3.6-27B and Qwen3.8-27B. The embedding, the output head, the MTP block and the linear weights take values in {โ1, 0, +1}; attention k/v and the linear-attention alpha/beta projections (191M parameters, about 0.7%) stay in F16. The model was trained with QAT against the merged model.
It thinks less per turn than Ternary-Bonsai-2-27B: on coding-agent tasks it reaches about the same score in about 60% of the time. This is an experimental model and trails Ternary-Bonsai-2-27B on some benchmarks.
Write-up: Experimenting with Ternary Quantization for LLMs (Qwen3.8-27B) (Japanese)
Files
All three GGUFs contain the same weights in different packings.
| File | Size | Runtime |
|---|---|---|
Sanshoku-Qwen3.8-3.6-27B-blend-Q2_0.gguf |
8.03 GB | upstream llama.cpp |
Sanshoku-Qwen3.8-3.6-27B-blend-PQ2_0.gguf |
7.61 GB | PrismML fork of llama.cpp |
Sanshoku-Qwen3.8-3.6-27B-blend-PTQ1_0.gguf |
6.34 GB | PrismML fork of llama.cpp |
mmproj-F16.gguf |
0.93 GB | vision encoder (merge of both models' encoders, FP16) |
Quick start
llama-server -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF -hff Sanshoku-Qwen3.8-3.6-27B-blend-Q2_0.gguf \
-ngl 99 -c 32768 -fa on --jinja \
--spec-type draft-mtp --spec-draft-n-max 2 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0
- Pick the file with
-hff(the tag:Q2_0can match the PQ2_0 file) - Tested with llama.cpp
f2cc728and the PrismML forkbdc23b5
Speed
Measured on an RTX 4000 Ada (20 GB). Prefill and decode are llama-bench pp512 and tg128. Decode + MTP is the average over four chat requests (256 output tokens, 2 draft tokens).
| Format | Runtime | Prefill (tok/s) โ | Decode (tok/s) โ | Decode + MTP (tok/s) โ |
|---|---|---|---|---|
| Q2_0 | upstream | 1117 | 35.4 | 56.8 |
| PQ2_0 | fork | 1027 | 37.8 | 54.9 |
| PTQ1_0 | fork | 1002 | 40.4 | 39.5 |
| Qwen3.8-27B UD-Q4_K_M (16.5 GB) | upstream | 985 | 19.3 | 34.8 |
Evaluation
Measured by us with one harness for all three models: swallow-evaluation-instruct (lighteval) and EvalScope, served by vLLM. Reasoning effort low, T 0.6 / top-p 0.95 / top-k 20, thinking on. โ higher is better, โ lower is better.
| Benchmark | Sanshoku-Qwen3.8-3.6-27B-blend | Ternary-Bonsai-2-27B | Qwen3.8-27B BF16 | |
|---|---|---|---|---|
| Size โ | 7.6 GB | 7.2 GB | 54 GB | |
| Agent | SWE-bench Verified (500) ยน โ | 58.0 | 60.8 ยฒ | 80.6 ยฒ |
| SWE-bench Verified Mini (50) ยณ โ | 58 | 60 | โ | |
| BFCL v3 โ | 75.0 | 73.9 | 76.0 | |
| English | MATH-500 โ | 89.0 | 98.6 | 99.0 |
| AIME 2024+2025 (pass@1, 4 samples) โ | 35.0 | 87.5 | 82.9 | |
| GPQA Diamond โ | 54.0 | 80.8 | 84.8 | |
| MMLU-Pro (1,001 sampled) โ | 71.0 | 80.4 | 82.6 | |
| LiveCodeBench v6 โ | 38.3 | 47.6 | 55.3 | |
| HumanEval+ โ | 86.7 | 92.7 | 93.4 | |
| IFEval (prompt strict) โ | 80.0 | 90.8 | 87.4 | |
| IFBench โ | 36.7 | 63.0 | 50.3 | |
| Japanese | JMMLU (1,060 sampled) โ | 80.7 | 86.3 | 89.7 |
| MMLU-ProX ja (1,001 sampled) โ | 63.6 | 75.5 | 79.6 | |
| JEMHopQA โ | 53.0 | 49.7 | 67.5 | |
| JGPQA Diamond โ | 52.0 | 77.8 | 83.8 | |
| MATH-100 ja โ | 81.8 | 96.0 | 96.0 | |
| JHumanEval โ | 85.2 | 92.9 | 92.9 | |
| M-IFEval ja โ | 53.5 | 62.8 | 65.7 | |
| WMT20 en-ja / ja-en (BLEU) โ | 22.5 / 21.9 | 23.1 / 22.5 | 22.8 / 23.4 | |
| MT-Bench ja โด โ | 5.90 | 6.19 | 7.07 |
- mini-SWE-agent 2.4.6, 250 steps, 3 h limit per instance
- Published in the Ternary Bonsai 2 whitepaper (a different harness)
- SWE-bench Verified Mini with mini-SWE-agent, 1 h limit per instance
- Swallow's Japanese MT-Bench questions and judge prompts, judged by Claude Sonnet 5.5 (1-10)
Training
- Each weight initialized to the ternary level nearest to the interval between its Qwen3.6 and Qwen3.8 values, in groups of 128 weights
- Trained with a straight-through estimator against the top-32 token distribution of the merged model
- About 1B tokens of training data
License
Apache-2.0 (see LICENSE). Derived from Qwen3.6-27B and Qwen3.8-27B.
An independent community model; not an official Qwen release.
Acknowledgements
Thanks to the Qwen team for the base models, JetBrains for the idea of merging the two (Qwen3.8-3.6-27B-blend), PrismML for the PQ2_0 and PTQ1_0 kernels, and the llama.cpp project.
๐ก Sanshoku-Qwen3.8-3.6-27B-blend-GGUF
Qwen3.6-27B ใจ Qwen3.8-27B ใใใผใธใใใขใใซใใ่ธ็ใใ3ๅคใขใใซใงใใembeddingใๅบๅ headใMTP ใใญใใฏใ็ทๅฝขๅฑคใฎ้ใฟใ {โ1, 0, +1} ใฎๅคใใจใใพใใattention ใฎ k/v ใจ linear attention ใฎ alpha/beta ๅฐๅฝฑ (1.91 ๅใใฉใกใผใฟใ็ด 0.7%) ใฏ F16 ใฎใพใพใงใใใใผใธใใใขใใซใๆๅธซใจใใฆ QAT ใงๅญฆ็ฟใใพใใใ
Ternary-Bonsai-2-27B ใใ1ใฟใผใณใใใใฎๆ่ใ็ญใใใณใผใใฃใณใฐใจใผใธใงใณใใฎใฟในใฏใงใปใผๅใๆญฃ่งฃ็ใซ็ด 6 ๅฒใฎๆ้ใงๅฐ้ใใพใใ ๅฎ้จ็ใชใขใใซใงใใใTernary-Bonsai-2-27B ใซๆฏในไธ้จใใณใใใผใฏใงๅฃใใพใใ
่งฃ่ชฌ่จไบ: LLM ใฎ3ๅค้ๅญๅใ่ฉฆใใ (Qwen3.8-27B) (English)
ใใกใคใซ
3ใคใฎ GGUF ใฏๅใ้ใฟใ็ฐใชใๅฝขๅผใงๆ ผ็ดใใฆใใพใใ
| ใใกใคใซ | ใตใคใบ | ๅฎ่ก็ฐๅข |
|---|---|---|
Sanshoku-Qwen3.8-3.6-27B-blend-Q2_0.gguf |
8.03 GB | ้ๅธธใฎ llama.cpp |
Sanshoku-Qwen3.8-3.6-27B-blend-PQ2_0.gguf |
7.61 GB | PrismML ็ llama.cpp (fork) |
Sanshoku-Qwen3.8-3.6-27B-blend-PTQ1_0.gguf |
6.34 GB | PrismML ็ llama.cpp (fork) |
mmproj-F16.gguf |
0.93 GB | ็ปๅใจใณใณใผใ (ไธกใขใใซใฎใจใณใณใผใใใใผใธใFP16) |
ไฝฟใๆน
llama-server -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF -hff Sanshoku-Qwen3.8-3.6-27B-blend-Q2_0.gguf \
-ngl 99 -c 32768 -fa on --jinja \
--spec-type draft-mtp --spec-draft-n-max 2 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0
- ใใกใคใซใฏ
-hffใงๆๅฎใใฆใใ ใใ (ใฟใฐ:Q2_0ใงใฏ PQ2_0 ใ้ธใฐใใใใจใใใใพใ) - llama.cpp
f2cc728ใจ PrismML ็bdc23b5ใงๅไฝ็ขบ่ชๆธใฟใงใ
้ๅบฆ
RTX 4000 Ada (20 GB) ใงๆธฌๅฎใprefill ใจ decode ใฏ llama-bench ใฎ pp512 ใจ tg128ใdecode + MTP ใฏ4ใคใฎใใฃใใใชใฏใจในใใฎๅนณๅใงใ (ๅบๅ 256 ใใผใฏใณใใใฉใใ 2 ใใผใฏใณ)ใ
| ๅฝขๅผ | ๅฎ่ก็ฐๅข | prefill (tok/s) โ | decode (tok/s) โ | decode + MTP (tok/s) โ |
|---|---|---|---|---|
| Q2_0 | ้ๅธธ็ | 1117 | 35.4 | 56.8 |
| PQ2_0 | fork | 1027 | 37.8 | 54.9 |
| PTQ1_0 | fork | 1002 | 40.4 | 39.5 |
| Qwen3.8-27B UD-Q4_K_M (16.5 GB) | ้ๅธธ็ | 985 | 19.3 | 34.8 |
่ฉไพก
3ใคใฎใขใใซใๅใ็ฐๅขใงๆธฌๅฎใใพใใใswallow-evaluation-instruct (lighteval) ใจ EvalScope ใไฝฟใใvLLM ใงๆจ่ซใใฆใใพใใreasoning effort lowใT 0.6 / top-p 0.95 / top-k 20ใthinking ใใใงใใโ ใฏ้ซใใปใฉ่ฏใใโ ใฏไฝใใปใฉ่ฏใๆๆจใงใใ
| ใใณใใใผใฏ | Sanshoku-Qwen3.8-3.6-27B-blend | Ternary-Bonsai-2-27B | Qwen3.8-27B BF16 | |
|---|---|---|---|---|
| ใตใคใบ โ | 7.6 GB | 7.2 GB | 54 GB | |
| ใจใผใธใงใณใ | SWE-bench Verified (500) ยน โ | 58.0 | 60.8 ยฒ | 80.6 ยฒ |
| SWE-bench Verified Mini (50) ยณ โ | 58 | 60 | โ | |
| BFCL v3 โ | 75.0 | 73.9 | 76.0 | |
| ่ฑ่ช | MATH-500 โ | 89.0 | 98.6 | 99.0 |
| AIME 2024+2025 (pass@1ใ4 ใตใณใใซ) โ | 35.0 | 87.5 | 82.9 | |
| GPQA Diamond โ | 54.0 | 80.8 | 84.8 | |
| MMLU-Pro (1,001 ๅใๆฝๅบ) โ | 71.0 | 80.4 | 82.6 | |
| LiveCodeBench v6 โ | 38.3 | 47.6 | 55.3 | |
| HumanEval+ โ | 86.7 | 92.7 | 93.4 | |
| IFEval (prompt strict) โ | 80.0 | 90.8 | 87.4 | |
| IFBench โ | 36.7 | 63.0 | 50.3 | |
| ๆฅๆฌ่ช | JMMLU (1,060 ๅใๆฝๅบ) โ | 80.7 | 86.3 | 89.7 |
| MMLU-ProX ja (1,001 ๅใๆฝๅบ) โ | 63.6 | 75.5 | 79.6 | |
| JEMHopQA โ | 53.0 | 49.7 | 67.5 | |
| JGPQA Diamond โ | 52.0 | 77.8 | 83.8 | |
| MATH-100 ja โ | 81.8 | 96.0 | 96.0 | |
| JHumanEval โ | 85.2 | 92.9 | 92.9 | |
| M-IFEval ja โ | 53.5 | 62.8 | 65.7 | |
| WMT20 en-ja / ja-en (BLEU) โ | 22.5 / 21.9 | 23.1 / 22.5 | 22.8 / 23.4 | |
| MT-Bench ja โด โ | 5.90 | 6.19 | 7.07 |
- mini-SWE-agent 2.4.6ใ250 ในใใใใ1ๅใใใ 3 ๆ้ๅถ้
- Ternary Bonsai 2 ใฎ whitepaper ใฎๅ ฌ่กจๅค (่ฉไพก็ฐๅขใ็ฐใชใใพใ)
- SWE-bench Verified Mini ใ mini-SWE-agent ใงๅฎ่กใ1ๅใใใ 1 ๆ้ๅถ้
- Swallow ใฎๆฅๆฌ่ช MT-Bench ใฎ่ณชๅใจๆก็นใใญใณใใใไฝฟใใClaude Sonnet 5.5 ใๆก็น (1ใ10)
ๅญฆ็ฟ
- ๅ้ใฟใ Qwen3.6 ใจ Qwen3.8 ใฎๅคใไธก็ซฏใจใใๅบ้ใซๆใ่ฟใ3ๅคใซ 128 ๅใใคใฎใฐใซใผใใงไธธใใฆๅๆๅ
- ใใผใธใใใขใใซใฎไธไฝ 32 ใใผใฏใณใฎๅๅธใ็ฎๆจใซ straight-through estimator ใงๅญฆ็ฟ
- ๅญฆ็ฟใใผใฟใฏ็ด 10 ๅใใผใฏใณ
ใฉใคใปใณใน
Apache-2.0 (LICENSE ใๅ็
ง)ใQwen3.6-27B ใจ Qwen3.8-27B ใฎๆดพ็็ฉใงใใ
ๅไบบใไฝๆใใๆดพ็ใขใใซใงใQwen ใฎๅ
ฌๅผใชใชใผในใงใฏใใใพใใใ
่ฌ่พ
ใใผในใขใใซใๅ ฌ้ใใฆใใ Qwen ใใผใ ใ2ใคใฎใขใใซใใใผใธใใใขใคใใขใฎๅ ใซใชใฃใ JetBrains (Qwen3.8-3.6-27B-blend)ใPQ2_0 ใจ PTQ1_0 ใฎใซใผใใซใ้็บใใ PrismMLใllama.cpp ใใญใธใงใฏใใซ ๆ่ฌใใพใใ
- Downloads last month
- 140
1-bit
2-bit
