๐Ÿก Sanshoku-Qwen3.8-3.6-27B-blend-GGUF

English | ๆ—ฅๆœฌ่ชž

A ternary model distilled from a merge of Qwen3.6-27B and Qwen3.8-27B. The embedding, the output head, the MTP block and the linear weights take values in {โˆ’1, 0, +1}; attention k/v and the linear-attention alpha/beta projections (191M parameters, about 0.7%) stay in F16. The model was trained with QAT against the merged model.

It thinks less per turn than Ternary-Bonsai-2-27B: on coding-agent tasks it reaches about the same score in about 60% of the time. This is an experimental model and trails Ternary-Bonsai-2-27B on some benchmarks.

Write-up: Experimenting with Ternary Quantization for LLMs (Qwen3.8-27B) (Japanese)

Files

All three GGUFs contain the same weights in different packings.

File Size Runtime
Sanshoku-Qwen3.8-3.6-27B-blend-Q2_0.gguf 8.03 GB upstream llama.cpp
Sanshoku-Qwen3.8-3.6-27B-blend-PQ2_0.gguf 7.61 GB PrismML fork of llama.cpp
Sanshoku-Qwen3.8-3.6-27B-blend-PTQ1_0.gguf 6.34 GB PrismML fork of llama.cpp
mmproj-F16.gguf 0.93 GB vision encoder (merge of both models' encoders, FP16)

Quick start

llama-server -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF -hff Sanshoku-Qwen3.8-3.6-27B-blend-Q2_0.gguf \
  -ngl 99 -c 32768 -fa on --jinja \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0
  • Pick the file with -hff (the tag :Q2_0 can match the PQ2_0 file)
  • Tested with llama.cpp f2cc728 and the PrismML fork bdc23b5

Speed

Measured on an RTX 4000 Ada (20 GB). Prefill and decode are llama-bench pp512 and tg128. Decode + MTP is the average over four chat requests (256 output tokens, 2 draft tokens).

Format Runtime Prefill (tok/s) โ†‘ Decode (tok/s) โ†‘ Decode + MTP (tok/s) โ†‘
Q2_0 upstream 1117 35.4 56.8
PQ2_0 fork 1027 37.8 54.9
PTQ1_0 fork 1002 40.4 39.5
Qwen3.8-27B UD-Q4_K_M (16.5 GB) upstream 985 19.3 34.8

Evaluation

Measured by us with one harness for all three models: swallow-evaluation-instruct (lighteval) and EvalScope, served by vLLM. Reasoning effort low, T 0.6 / top-p 0.95 / top-k 20, thinking on. โ†‘ higher is better, โ†“ lower is better.

Benchmark Sanshoku-Qwen3.8-3.6-27B-blend Ternary-Bonsai-2-27B Qwen3.8-27B BF16
Size โ†“ 7.6 GB 7.2 GB 54 GB
Agent SWE-bench Verified (500) ยน โ†‘ 58.0 60.8 ยฒ 80.6 ยฒ
SWE-bench Verified Mini (50) ยณ โ†‘ 58 60 โ€“
BFCL v3 โ†‘ 75.0 73.9 76.0
English MATH-500 โ†‘ 89.0 98.6 99.0
AIME 2024+2025 (pass@1, 4 samples) โ†‘ 35.0 87.5 82.9
GPQA Diamond โ†‘ 54.0 80.8 84.8
MMLU-Pro (1,001 sampled) โ†‘ 71.0 80.4 82.6
LiveCodeBench v6 โ†‘ 38.3 47.6 55.3
HumanEval+ โ†‘ 86.7 92.7 93.4
IFEval (prompt strict) โ†‘ 80.0 90.8 87.4
IFBench โ†‘ 36.7 63.0 50.3
Japanese JMMLU (1,060 sampled) โ†‘ 80.7 86.3 89.7
MMLU-ProX ja (1,001 sampled) โ†‘ 63.6 75.5 79.6
JEMHopQA โ†‘ 53.0 49.7 67.5
JGPQA Diamond โ†‘ 52.0 77.8 83.8
MATH-100 ja โ†‘ 81.8 96.0 96.0
JHumanEval โ†‘ 85.2 92.9 92.9
M-IFEval ja โ†‘ 53.5 62.8 65.7
WMT20 en-ja / ja-en (BLEU) โ†‘ 22.5 / 21.9 23.1 / 22.5 22.8 / 23.4
MT-Bench ja โด โ†‘ 5.90 6.19 7.07
  1. mini-SWE-agent 2.4.6, 250 steps, 3 h limit per instance
  2. Published in the Ternary Bonsai 2 whitepaper (a different harness)
  3. SWE-bench Verified Mini with mini-SWE-agent, 1 h limit per instance
  4. Swallow's Japanese MT-Bench questions and judge prompts, judged by Claude Sonnet 5.5 (1-10)

Training

  • Each weight initialized to the ternary level nearest to the interval between its Qwen3.6 and Qwen3.8 values, in groups of 128 weights
  • Trained with a straight-through estimator against the top-32 token distribution of the merged model
  • About 1B tokens of training data

License

Apache-2.0 (see LICENSE). Derived from Qwen3.6-27B and Qwen3.8-27B. An independent community model; not an official Qwen release.

Acknowledgements

Thanks to the Qwen team for the base models, JetBrains for the idea of merging the two (Qwen3.8-3.6-27B-blend), PrismML for the PQ2_0 and PTQ1_0 kernels, and the llama.cpp project.


๐Ÿก Sanshoku-Qwen3.8-3.6-27B-blend-GGUF

Qwen3.6-27B ใจ Qwen3.8-27B ใ‚’ใƒžใƒผใ‚ธใ—ใŸใƒขใƒ‡ใƒซใ‹ใ‚‰่’ธ็•™ใ—ใŸ3ๅ€คใƒขใƒ‡ใƒซใงใ™ใ€‚embeddingใ€ๅ‡บๅŠ› headใ€MTP ใƒ–ใƒญใƒƒใ‚ฏใ€็ทšๅฝขๅฑคใฎ้‡ใฟใŒ {โˆ’1, 0, +1} ใฎๅ€คใ‚’ใจใ‚Šใพใ™ใ€‚attention ใฎ k/v ใจ linear attention ใฎ alpha/beta ๅฐ„ๅฝฑ (1.91 ๅ„„ใƒ‘ใƒฉใƒกใƒผใ‚ฟใ€็ด„ 0.7%) ใฏ F16 ใฎใพใพใงใ™ใ€‚ใƒžใƒผใ‚ธใ—ใŸใƒขใƒ‡ใƒซใ‚’ๆ•™ๅธซใจใ—ใฆ QAT ใงๅญฆ็ฟ’ใ—ใพใ—ใŸใ€‚

Ternary-Bonsai-2-27B ใ‚ˆใ‚Š1ใ‚ฟใƒผใƒณใ‚ใŸใ‚Šใฎๆ€่€ƒใŒ็Ÿญใใ€ใ‚ณใƒผใƒ‡ใ‚ฃใƒณใ‚ฐใ‚จใƒผใ‚ธใ‚งใƒณใƒˆใฎใ‚ฟใ‚นใ‚ฏใงใปใผๅŒใ˜ๆญฃ่งฃ็އใซ็ด„ 6 ๅ‰ฒใฎๆ™‚้–“ใงๅˆฐ้”ใ—ใพใ™ใ€‚ ๅฎŸ้จ“็š„ใชใƒขใƒ‡ใƒซใงใ‚ใ‚Šใ€Ternary-Bonsai-2-27B ใซๆฏ”ในไธ€้ƒจใƒ™ใƒณใƒใƒžใƒผใ‚ฏใงๅŠฃใ‚Šใพใ™ใ€‚

่งฃ่ชฌ่จ˜ไบ‹: LLM ใฎ3ๅ€ค้‡ๅญๅŒ–ใ‚’่ฉฆใ—ใŸ (Qwen3.8-27B) (English)

ใƒ•ใ‚กใ‚คใƒซ

3ใคใฎ GGUF ใฏๅŒใ˜้‡ใฟใ‚’็•ฐใชใ‚‹ๅฝขๅผใงๆ ผ็ดใ—ใฆใ„ใพใ™ใ€‚

ใƒ•ใ‚กใ‚คใƒซ ใ‚ตใ‚คใ‚บ ๅฎŸ่กŒ็’ฐๅขƒ
Sanshoku-Qwen3.8-3.6-27B-blend-Q2_0.gguf 8.03 GB ้€šๅธธใฎ llama.cpp
Sanshoku-Qwen3.8-3.6-27B-blend-PQ2_0.gguf 7.61 GB PrismML ็‰ˆ llama.cpp (fork)
Sanshoku-Qwen3.8-3.6-27B-blend-PTQ1_0.gguf 6.34 GB PrismML ็‰ˆ llama.cpp (fork)
mmproj-F16.gguf 0.93 GB ็”ปๅƒใ‚จใƒณใ‚ณใƒผใƒ€ (ไธกใƒขใƒ‡ใƒซใฎใ‚จใƒณใ‚ณใƒผใƒ€ใ‚’ใƒžใƒผใ‚ธใ€FP16)

ไฝฟใ„ๆ–น

llama-server -hf oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF -hff Sanshoku-Qwen3.8-3.6-27B-blend-Q2_0.gguf \
  -ngl 99 -c 32768 -fa on --jinja \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0
  • ใƒ•ใ‚กใ‚คใƒซใฏ -hff ใงๆŒ‡ๅฎšใ—ใฆใใ ใ•ใ„ (ใ‚ฟใ‚ฐ :Q2_0 ใงใฏ PQ2_0 ใŒ้ธใฐใ‚Œใ‚‹ใ“ใจใŒใ‚ใ‚Šใพใ™)
  • llama.cpp f2cc728 ใจ PrismML ็‰ˆ bdc23b5 ใงๅ‹•ไฝœ็ขบ่ชๆธˆใฟใงใ™

้€Ÿๅบฆ

RTX 4000 Ada (20 GB) ใงๆธฌๅฎšใ€‚prefill ใจ decode ใฏ llama-bench ใฎ pp512 ใจ tg128ใ€decode + MTP ใฏ4ใคใฎใƒใƒฃใƒƒใƒˆใƒชใ‚ฏใ‚จใ‚นใƒˆใฎๅนณๅ‡ใงใ™ (ๅ‡บๅŠ› 256 ใƒˆใƒผใ‚ฏใƒณใ€ใƒ‰ใƒฉใƒ•ใƒˆ 2 ใƒˆใƒผใ‚ฏใƒณ)ใ€‚

ๅฝขๅผ ๅฎŸ่กŒ็’ฐๅขƒ prefill (tok/s) โ†‘ decode (tok/s) โ†‘ decode + MTP (tok/s) โ†‘
Q2_0 ้€šๅธธ็‰ˆ 1117 35.4 56.8
PQ2_0 fork 1027 37.8 54.9
PTQ1_0 fork 1002 40.4 39.5
Qwen3.8-27B UD-Q4_K_M (16.5 GB) ้€šๅธธ็‰ˆ 985 19.3 34.8

่ฉ•ไพก

3ใคใฎใƒขใƒ‡ใƒซใ‚’ๅŒใ˜็’ฐๅขƒใงๆธฌๅฎšใ—ใพใ—ใŸใ€‚swallow-evaluation-instruct (lighteval) ใจ EvalScope ใ‚’ไฝฟใ„ใ€vLLM ใงๆŽจ่ซ–ใ—ใฆใ„ใพใ™ใ€‚reasoning effort lowใ€T 0.6 / top-p 0.95 / top-k 20ใ€thinking ใ‚ใ‚Šใงใ™ใ€‚โ†‘ ใฏ้ซ˜ใ„ใปใฉ่‰ฏใใ€โ†“ ใฏไฝŽใ„ใปใฉ่‰ฏใ„ๆŒ‡ๆจ™ใงใ™ใ€‚

ใƒ™ใƒณใƒใƒžใƒผใ‚ฏ Sanshoku-Qwen3.8-3.6-27B-blend Ternary-Bonsai-2-27B Qwen3.8-27B BF16
ใ‚ตใ‚คใ‚บ โ†“ 7.6 GB 7.2 GB 54 GB
ใ‚จใƒผใ‚ธใ‚งใƒณใƒˆ SWE-bench Verified (500) ยน โ†‘ 58.0 60.8 ยฒ 80.6 ยฒ
SWE-bench Verified Mini (50) ยณ โ†‘ 58 60 โ€“
BFCL v3 โ†‘ 75.0 73.9 76.0
่‹ฑ่ชž MATH-500 โ†‘ 89.0 98.6 99.0
AIME 2024+2025 (pass@1ใ€4 ใ‚ตใƒณใƒ—ใƒซ) โ†‘ 35.0 87.5 82.9
GPQA Diamond โ†‘ 54.0 80.8 84.8
MMLU-Pro (1,001 ๅ•ใ‚’ๆŠฝๅ‡บ) โ†‘ 71.0 80.4 82.6
LiveCodeBench v6 โ†‘ 38.3 47.6 55.3
HumanEval+ โ†‘ 86.7 92.7 93.4
IFEval (prompt strict) โ†‘ 80.0 90.8 87.4
IFBench โ†‘ 36.7 63.0 50.3
ๆ—ฅๆœฌ่ชž JMMLU (1,060 ๅ•ใ‚’ๆŠฝๅ‡บ) โ†‘ 80.7 86.3 89.7
MMLU-ProX ja (1,001 ๅ•ใ‚’ๆŠฝๅ‡บ) โ†‘ 63.6 75.5 79.6
JEMHopQA โ†‘ 53.0 49.7 67.5
JGPQA Diamond โ†‘ 52.0 77.8 83.8
MATH-100 ja โ†‘ 81.8 96.0 96.0
JHumanEval โ†‘ 85.2 92.9 92.9
M-IFEval ja โ†‘ 53.5 62.8 65.7
WMT20 en-ja / ja-en (BLEU) โ†‘ 22.5 / 21.9 23.1 / 22.5 22.8 / 23.4
MT-Bench ja โด โ†‘ 5.90 6.19 7.07
  1. mini-SWE-agent 2.4.6ใ€250 ใ‚นใƒ†ใƒƒใƒ—ใ€1ๅ•ใ‚ใŸใ‚Š 3 ๆ™‚้–“ๅˆถ้™
  2. Ternary Bonsai 2 ใฎ whitepaper ใฎๅ…ฌ่กจๅ€ค (่ฉ•ไพก็’ฐๅขƒใŒ็•ฐใชใ‚Šใพใ™)
  3. SWE-bench Verified Mini ใ‚’ mini-SWE-agent ใงๅฎŸ่กŒใ€1ๅ•ใ‚ใŸใ‚Š 1 ๆ™‚้–“ๅˆถ้™
  4. Swallow ใฎๆ—ฅๆœฌ่ชž MT-Bench ใฎ่ณชๅ•ใจๆŽก็‚นใƒ—ใƒญใƒณใƒ—ใƒˆใ‚’ไฝฟใ„ใ€Claude Sonnet 5.5 ใŒๆŽก็‚น (1ใ€œ10)

ๅญฆ็ฟ’

  • ๅ„้‡ใฟใ‚’ Qwen3.6 ใจ Qwen3.8 ใฎๅ€คใ‚’ไธก็ซฏใจใ™ใ‚‹ๅŒบ้–“ใซๆœ€ใ‚‚่ฟ‘ใ„3ๅ€คใซ 128 ๅ€‹ใšใคใฎใ‚ฐใƒซใƒผใƒ—ใงไธธใ‚ใฆๅˆๆœŸๅŒ–
  • ใƒžใƒผใ‚ธใ—ใŸใƒขใƒ‡ใƒซใฎไธŠไฝ 32 ใƒˆใƒผใ‚ฏใƒณใฎๅˆ†ๅธƒใ‚’็›ฎๆจ™ใซ straight-through estimator ใงๅญฆ็ฟ’
  • ๅญฆ็ฟ’ใƒ‡ใƒผใ‚ฟใฏ็ด„ 10 ๅ„„ใƒˆใƒผใ‚ฏใƒณ

ใƒฉใ‚คใ‚ปใƒณใ‚น

Apache-2.0 (LICENSE ใ‚’ๅ‚็…ง)ใ€‚Qwen3.6-27B ใจ Qwen3.8-27B ใฎๆดพ็”Ÿ็‰ฉใงใ™ใ€‚ ๅ€‹ไบบใŒไฝœๆˆใ—ใŸๆดพ็”Ÿใƒขใƒ‡ใƒซใงใ€Qwen ใฎๅ…ฌๅผใƒชใƒชใƒผใ‚นใงใฏใ‚ใ‚Šใพใ›ใ‚“ใ€‚

่ฌ่พž

ใƒ™ใƒผใ‚นใƒขใƒ‡ใƒซใ‚’ๅ…ฌ้–‹ใ—ใฆใ„ใ‚‹ Qwen ใƒใƒผใƒ ใ€2ใคใฎใƒขใƒ‡ใƒซใ‚’ใƒžใƒผใ‚ธใ™ใ‚‹ใ‚ขใ‚คใƒ‡ใ‚ขใฎๅ…ƒใซใชใฃใŸ JetBrains (Qwen3.8-3.6-27B-blend)ใ€PQ2_0 ใจ PTQ1_0 ใฎใ‚ซใƒผใƒใƒซใ‚’้–‹็™บใ—ใŸ PrismMLใ€llama.cpp ใƒ—ใƒญใ‚ธใ‚งใ‚ฏใƒˆใซ ๆ„Ÿ่ฌใ—ใพใ™ใ€‚

Downloads last month
140
GGUF
Model size
27B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

1-bit

2-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for oshiteku/Sanshoku-Qwen3.8-3.6-27B-blend-GGUF

Merge model
this model