A Multi-Token Prediction tune of Qwen3.6-27B fine-tuned for reasoning-aware agentic coding through a PI-style harness. Packaged as llama.cpp-compatible GGUF for local agent loops.
Multi-Token Prediction drafts future tokens and accepts them when the main decode path agrees — cutting wall time on long reasoning, code generation, and tool-call setup.
🧩
Reasoning by Design
Trained with reasoning-bearing PI traces for tasks that benefit from scratch work, while preserving the direct tool-call and edit discipline needed by local agent harnesses.
🧪
Local-First Throughput
Designed for llama.cpp-class runtimes on a single workstation, with MTP draft acceptance measured on real agent workloads.
🚀
Practical Deliberation
Built for workflows where the model should think when the task is hard, then converge into concrete tool calls, edits, commands, and structured answers.
Context Window
128k tested
256k natively supported by the base · extensible up to 1M tokens via RoPE scaling.
MTP Draft Acceptance
~78%
Drafted future tokens accepted by the main decode path on agent workloads. 3 speculative steps · 4 draft tokens.
## 💡 1. Model Overview
| Attribute | Details |
| --- | --- |
| Base model | [`Qwen/Qwen3.6-27B`](https://huggingface.co/Qwen/Qwen3.6-27B) |
| Release format | GGUF |
| Runtime target | llama.cpp-compatible local inference |
| Model capabilities | Text generation, reasoning, tool use / function calling, vision-language understanding, image-text-to-text with `mmproj-F16.gguf` |
| Tuning focus | Reasoning-aware harness fluency, coding-agent tasks, terminal workflows, tool use, repository work |
| Reasoning data policy | Reasoning-supervised trajectories are included for complex tasks; direct-action traces are preserved for agent loops |
| Recommended quant | `Q4_K_M` as the default starting point |
> [!NOTE]
> Qwen3.6-27B is a causal language model with a vision encoder. **Vision-language and image-text-to-text understanding are supported** in this release by pairing the language-model GGUF with the `mmproj-F16.gguf` sidecar mirrored in this repo (see §4 *Multi-modal inference*). This is image/video input to text output, not text-to-image generation. The **MTP draft heads are kept at `Q8_0` precision inside every quant** via `--tensor-type nextn=q8_0` at quantize time — speculative decoding works at any quant level, not just `Q8_0` / `bf16`.
## 🧩 2. Why This Tune
Qwen3.6 operates in **thinking mode by default**, emitting a `...` reasoning block before the final response, and can also be run in direct-response / non-thinking mode when a harness needs lower latency. This release is tuned to keep the reasoning path useful for complex coding and operations tasks instead of treating thinking tokens as generic verbosity.
The PI-style training mix emphasizes trajectories where reasoning, tool use, terminal feedback, repository edits, and verifier results work together. The goal is not just longer scratch work; it is better planning before edits, better recovery after failed commands, and cleaner handoff from deliberation into concrete action.
- Terminal and shell task execution.
- Repository inspection, patching, and test iteration.
- Tool-call-shaped interactions and structured outputs.
- DevOps runbooks, environment setup, and debugging loops.
- Coding tasks where command use, file edits, and verifier feedback matter.
- Reasoning-heavy tasks where preserving the chain of decisions improves the next step.
## 📦 3. Quantizations
Recommended starting point: `Q4_K_M`.
| Quant | File size | VRAM (approx) | Suggested use |
| --- | ---: | ---: | --- |
| `Q2_K` | ~11 GB | ~13 GB | Smallest memory footprint; quality tradeoffs are expected. |
| `Q3_K_S` | ~12 GB | ~15 GB | Low-memory 3-bit option. |
| `Q3_K_M` | ~14 GB | ~16 GB | Balanced 3-bit option. |
| `Q3_K_L` | ~15 GB | ~17 GB | Higher-quality 3-bit option. |
| `Q4_K_S` | ~16 GB | ~18 GB | Smaller 4-bit option. |
| `Q4_K_M` | ~17 GB | ~19 GB | **Default recommendation for most local use.** Comfortable on a 24 GB GPU. |
| `Q5_K_S` | ~19 GB | ~21 GB | Higher-quality 5-bit option. |
| `Q5_K_M` | ~20 GB | ~22 GB | Strong quality/memory tradeoff; near the upper edge of a 24 GB GPU. |
| `Q6_K` | ~22 GB | ~25 GB | High-quality local inference if you have the memory. |
| `Q8_0` | ~29 GB | ~32 GB | Highest-precision quantized option. |
| `bf16` | ~55 GB | ~58 GB | BF16 GGUF reference, if present. |
VRAM figures are rough estimates for GPU-offload inference (`-ngl 99 -fa`) at a moderate context (~32k) with quantized KV cache; they scale up with longer contexts.
> [!IMPORTANT]
> **Every quant in this release ships with the MTP `nextn` prediction heads stored at `Q8_0` precision**, regardless of the overall quant target. That means speculative decoding works at any quant level — pick the smallest one that fits your VRAM and you still get the MTP throughput profile described in §6.
Some files may still be uploading. Check the Files tab for the exact artifacts currently available.
## 🚀 4. Quickstart
Run with llama.cpp in the reasoning-oriented path (standard launch — works on any build):
```bash
# 128k context shown; base model natively supports 256k and is extensible to ~1M via RoPE scaling.
# Sampling values follow Qwen3.6's thinking-mode guidance for precise coding tasks.
llama-server -hf bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF:Q4_K_M \
--jinja -ngl 99 -fa -c 131072 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
--presence-penalty 0.0
```
Run with **upstream llama.cpp + MTP speculative decoding** ([`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp), MTP support merged in [PR #22673](https://github.com/ggml-org/llama.cpp/pull/22673)):
```bash
# The nextn prediction heads in this release activate via upstream's draft-mtp speculator.
# -np must be 1 with MTP (parallel slots are not yet supported alongside MTP).
llama-server -hf bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF:Q4_K_M \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
-np 1 \
--jinja -ngl 99 -fa -c 131072 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
--presence-penalty 0.0
```
For direct-response / non-thinking turns, use Qwen3.6's instruct-mode sampling instead:
```bash
llama-server -hf bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF:Q4_K_M \
--jinja -ngl 99 -fa -c 131072 \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 \
--presence-penalty 1.5
```
Run with Ollama:
```bash
ollama run hf.co/bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF:Q4_K_M
```
Download a single GGUF file:
```bash
hf download bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF \
Qwen3.6-27B-MTP-pi-reasoning-Q4_K_M.gguf \
--local-dir .
```
Download the whole repo:
```bash
hf download bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF --local-dir .
```
### Vision / image-text-to-text inference
This repo mirrors `mmproj-F16.gguf` (~928 MB), enabling vision-language / image-text-to-text inference out of the box — no second-source download required. **A single mmproj file pairs with every quant** in this release; the projector is architecturally tied to the base model's vision tower, not to the LM quant level, so download it once and reuse it.
The mmproj is mirrored from [`unsloth/Qwen3.6-27B-MTP-GGUF`](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF). The fine-tune in this release is **language-only** — the vision encoder weights have not been touched. Image / video understanding is therefore inherited unchanged from the upstream Qwen3.6-27B base model; this release does not claim to improve it, only to preserve it.
```bash
# Pulls both the LM weights and the mmproj sidecar in one go
hf download bytkim/Qwen3.6-27B-MTP-pi-reasoning-GGUF \
Qwen3.6-27B-MTP-pi-reasoning-Q4_K_M.gguf mmproj-F16.gguf \
--local-dir .
# Launch llama-server with vision attached
llama-server -m ./Qwen3.6-27B-MTP-pi-reasoning-Q4_K_M.gguf \
--mmproj ./mmproj-F16.gguf \
--jinja -ngl 99 -fa -c 131072 \
--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
--presence-penalty 0.0
```
For a quick text-and-image session without spinning up a server:
```bash
llama-mtmd-cli -m ./Qwen3.6-27B-MTP-pi-reasoning-Q4_K_M.gguf \
--mmproj ./mmproj-F16.gguf
```
### Use as an OpenAI-compatible API
`llama-server` exposes an OpenAI-compatible `/v1/chat/completions` endpoint, so any client written against the OpenAI SDK can point at it directly — no client changes required:
```python
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8080/v1",
api_key="not-needed",
)
resp = client.chat.completions.create(
model="Qwen3.6-27B-MTP-pi-reasoning",
messages=[
{"role": "system", "content": "You are a precise coding agent."},
{"role": "user", "content": "Write a Python function that merges overlapping intervals."},
],
)
print(resp.choices[0].message.content)
```
The same endpoint accepts `tools=[...]` for function calling and supports streaming via `stream=True`.
Qwen3.6's thinking mode is the default path. For direct non-thinking responses in API runtimes that expose chat-template options, set `chat_template_kwargs={"enable_thinking": False}` on the request body. For long-running agent sessions where the runtime supports it, `chat_template_kwargs={"preserve_thinking": True}` can preserve earlier reasoning context across turns.
## 🧬 5. Training & Data Notes
Tuned on **real agent traces**, not synthetic generations or distilled chat — every training row is the trail of an assistant that actually executed the task end-to-end through a PI-style harness, exported as Qwen-compatible ChatML rows with tool schemas and runtime prompts preserved. This reasoning release keeps reasoning-bearing supervision for tasks where the plan and verifier loop matter, while retaining direct tool and edit behavior for harness compatibility.
High-level task coverage spanned:
- Terminal and shell-environment agent tasks.
- Tool / function-calling interactions.
- Multi-language code editing and repair tasks.
- Repository issue resolution and test-driven patching.
- Coding and API integration tasks.
- Shell, package, migration, ops, and verifier-driven tasks.
🧭
Training philosophy
Full supervised fine-tune (no LoRA). This release keeps reasoning supervision for complex tasks and tool-feedback loops, rather than stripping the trace down to direct answers only. It is intended to be strongest in Qwen3.6's thinking-capable path, with direct-response mode still available when latency matters.
Specific dataset names and training-row counts are intentionally omitted from this initial card.
## ⚡ 6. MTP Throughput
MTP stands for **Multi-Token Prediction**. The model drafts likely future tokens and the runtime accepts them when they agree with the main decode path. On local agent work this matters because long reasoning, code generation, tool-call setup, and shell-oriented turns otherwise spend most of their wall time waiting on generation.
The numbers below are carried forward as the expected local MTP profile for this release family and should be re-measured for the final reasoning quant files. Full benchmark publication is forthcoming and will replace these with task-success-rate tables in §7.
### Raw decode profile
Prompt / Prefill
~615 tok/s
Reading and processing prompt / context tokens.
Decode / Generation
~40 tok/s
Raw generated-token speed reported by llama.cpp.
End-to-End Request
~71 tok/s
Full llama request throughput across prompt and decode.
MTP Draft Acceptance
~78%
Share of drafted future tokens accepted by the main decode path.
### Agentic throughput
The agentic number is different from raw tokens/sec. It measures real task throughput across agent runs — including model generation, tool calls, shell commands, package installs, file I/O, and verifier-facing work.
Effective Output
~33 tok/s
Output tokens divided by full agent execution time.
Effective Total
~1.6k tok/s
Input + output tokens counted by the harness, end-to-end.
Effective output throughput is computed as:
```text
sum(output tokens) / sum(agent execution duration)
```
That makes it a more realistic agent-workflow number than plain decode speed — it includes time spent operating through the harness, not just time spent generating text.
## 📊 7. Coding Eval Benchmarks
🛰️
Releasing soon
Task-success and reasoning-quality benchmarks are tracked separately from throughput and will be published in a follow-up update to this card. Throughput answers how fast the local MTP stack moves; evals answer how often the agent actually solves the task — the two should not be inferred from one another.
The follow-up release will cover task-success rates across the high-level areas listed in §5: terminal/shell agent tasks, tool & function calling, multi-language code editing, repository issue resolution, coding/API integration tasks, and reasoning-heavy repair loops.
> [!IMPORTANT]
> Throughput figures above are from the local MTP-enabled release-family run. Task success rates should be reported only from completed eval runs, not inferred from speed.
## 🎯 8. Recommended Use Cases
- Local coding-agent experiments.
- Tool-heavy chat and function-calling experiments.
- DevOps troubleshooting and runbook drafting.
- Repository navigation, patch planning, and test iteration.
- Long-context engineering workflows where local inference is preferred.
- Reasoning-heavy debugging, architecture review, and verifier-guided repair loops.
## ⚠️ 9. Limitations
- This is a community release for research, evaluation, and workflow exploration.
- Low-bit quantizations may reduce instruction following and tool-call reliability.
- Reasoning mode emits extra tokens before the final response; use direct-response mode when latency is more important than deliberation.
- Coding-eval success rates are not finalized in this initial card.
- This card does not claim safety alignment beyond the behavior inherited from the base model and the fine-tuning data.
## 📜 10. License
Released under the [Apache 2.0](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE) license, inherited from the upstream Qwen3.6-27B base model. You are free to use, modify, and redistribute the model and its derivatives subject to the terms of that license.
## 🙏 11. Acknowledgements
Thanks to the **Qwen team** for the Qwen3.6 base model and its MTP design, to the **ggml-org / llama.cpp** maintainers for native multi-token-prediction support in upstream, and to the broader open-source quantization tooling community whose work makes local-first inference of frontier models possible.
Built for local agent loops
Reasoning where it helps · speed where it matters · MTP at every quant.