nerkyor

/

Qwen3.8-27B-Coder390-EfficientThink-Opus5.5-GPT6Astra-Grok4.7-DSV4Pro-K3-SFT-RLOO-GGUF-NInfer

Like 69

That FATAL is the engine, not a broken Q6_K file — unknown tensor layout: gguf_blocks_v1 means the binary you're running can't read GGUF-block .ninfer packs.

Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer is loadable only by NInfer-all (iamwavecut/ninfer-all, master). Official ninfer-src / stock container builds cannot read gguf_blocks_v1 (same note is on the model card). Swap the image/binary to NInfer-all and keep the same weights; your flags look fine for a first smoke test (--spec mtp, concurrency 1).

If you just need it running meanwhile, the matching GGUF in this repo works on upstream llama.cpp with built-in MTP: --spec-type draft-mtp --spec-draft-n-max 4 (plus --mmproj if you want vision).

After switching to NInfer-all, if it still dies on startup, paste the new log and the exact commit/image tag you're on.

Here's ninfer-all output:

ninfer-1 | Qwen3.8-27B | C2 | context 131072 | KV pool 131072 | rk4v4 | MTP3 + draft head, Q6 head, two lanes | vision (overlay)
ninfer-1 | Prefill: cuBLAS route, chunk 2048
ninfer-1 | Cache: 8 shared / 8 private / 32 host states | automatic prefix grid on
ninfer-1 | API: http://0.0.0.0:8080/v1
ninfer-1 |
ninfer-1 | ninfer-serve: CUDA 13 backend
ninfer-1 | 2026-10-27 10:51:55.550 INFO build 2026-10-26.3+b96958a
ninfer-1 | 2026-10-27 10:51:55.565 INFO starting engine
ninfer-1 | 2026-10-27 10:51:55.915 INFO engine | device profile nvidia-geforce-rtx-5090-sm120: 62 routed keys (built in: RTX 5090 at 575 W, 2026-09-25; unified routes and two-stage GDN at 450 W, 2026-09-27; Q4/Q5 MoE prefill tile at 450 W, 2026-10-15)
ninfer-1 | 2026-10-27 10:51:57.147 ERROR startup failed | planning runtime | 1.2s
ninfer-1 | 2026-10-27 10:51:57.150 ERROR selected proposal head is absent from artifact
ninfer-1 exited with code 0

services:
ninfer:
image: ghcr.io/iamwavecut/ninfer-all:${NINFER_CUDA_TAG:-cuda13.4}
command: ["run", "qwen38-27b"]
ports:
- "11434:8080"
volumes:
- /ssd/ollama:/models # .ninfer artifacts
- ./grafts:/grafts # optional prompt grafts: grafts//NAME.bin + NAME.json
- ./ninfer-cache:/cache # device profiles from calibration
environment:
NINFER_SPEC: mtp
NINFER_CONTEXT: "131072"
NINFER_MODEL: "Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer"
NINFER_VISION: "on"

Good, you're on NInfer-all now — that clears the earlier gguf_blocks_v1 issue.

This new error is a different flag mismatch. selected proposal head is absent from artifact means the engine asked for the optimized proposal head (--lm-head-draft), but Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer only ships the built-in MTP head, not that proposal head. Your banner line MTP3 + draft head is the giveaway: something in the launch path is turning draft-head on.

For this package, use MTP without --lm-head-draft. Card-style smoke command:

ninfer-serve /models/Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer \
  --host 0.0.0.0 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-q6k-mtp \
  --max-context 102400 --kv-capacity 409600 --kv-dtype rk8v4 \
  --max-concurrency 4 --spec mtp --draft-tokens 3 --vision

If you're staying on the ninfer-all Compose helper (command: ["run", "qwen38-27b"] + NINFER_SPEC=mtp), check whether that preset injects --lm-head-draft and turn it off, or just call ninfer-serve with the flags above.

Also avoid --max-context 131072 --kv-capacity 131072 for this tier — that KV pool only holds one full context, so extra requests get HTTP 503. The card recommends 102400 / 409600 with concurrency 4 for Q6_K.

Fallback if you just need it up: the matching GGUF in this repo on upstream llama.cpp with built-in MTP: --spec-type draft-mtp --spec-draft-n-max 4 (plus --mmproj for vision).

If it still dies after dropping draft-head, paste the new startup log.

Thank you for your response. Much better now but still something is wrong

ninfer-1 | ninfer-serve: CUDA 13 backend
ninfer-1 | 2026-10-27 11:50:46.374 INFO build 2026-10-26.3+b96958a
ninfer-1 | 2026-10-27 11:50:46.391 INFO starting engine
ninfer-1 | 2026-10-27 11:50:46.775 INFO engine | device profile nvidia-geforce-rtx-5090-sm120: 62 routed keys (built in: RTX 5090 at 575 W, 2026-09-25; unified routes and two-stage GDN at 450 W, 2026-09-27; Q4/Q5 MoE prefill tile at 450 W, 2026-10-15)
ninfer-1 | 2026-10-27 11:50:47.948 INFO loading weights | 21.8 GiB
ninfer-1 | ninfer-serve: CUDA 13 backend
ninfer-1 | 2026-10-27 11:50:57.331 INFO build 2026-10-26.3+b96958a
ninfer-1 | 2026-10-27 11:50:57.344 INFO starting engine
ninfer-1 | 2026-10-27 11:50:57.709 INFO engine | device profile nvidia-geforce-rtx-5090-sm120: 62 routed keys (built in: RTX 5090 at 575 W, 2026-09-25; unified routes and two-stage GDN at 450 W, 2026-09-27; Q4/Q5 MoE prefill tile at 450 W, 2026-10-15)
ninfer-1 | 2026-10-27 11:50:58.945 INFO loading weights | 21.8 GiB
ninfer-1 | 2026-10-27 11:51:09.083 INFO loading weights 23.0% | 5.00 GiB/21.8 GiB | 402.7 MiB/s | ETA 42.6s
ninfer-1 | 2026-10-27 11:51:19.135 INFO loading weights 37.9% | 8.25 GiB/21.8 GiB | 410.3 MiB/s | ETA 33.7s
ninfer-1 | 2026-10-27 11:51:29.237 INFO loading weights 53.7% | 11.7 GiB/21.8 GiB | 407.3 MiB/s | ETA 25.3s
ninfer-1 | 2026-10-27 11:51:39.386 INFO loading weights 69.5% | 15.1 GiB/21.8 GiB | 402.2 MiB/s | ETA 16.9s
ninfer-1 | 2026-10-27 11:51:49.441 INFO loading weights 84.7% | 18.4 GiB/21.8 GiB | 406.0 MiB/s | ETA 8.4s
ninfer-1 | 2026-10-27 11:51:59.364 INFO weights ready | 21.8 GiB | 1m 0.4s | 368.8 MiB/s
ninfer-1 | 2026-10-27 11:51:59.527 ERROR startup failed | finalizing target | 154 ms
ninfer-1 | 2026-10-27 11:51:59.625 ERROR requested Engine runtime reservation requires 14769809665 bytes, but only 9806872576 bytes are available for runtime capacity

command:
- serve
- "/models/Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer"
- --model-id
- qwen3.8-27b
- --max-context
- "102400"
- --kv-capacity
- "409600"
- --kv-dtype
- fp8
- --preserve-thinking
- --vision
- --max-concurrency.
- "4"
- --kv-dtype
- rk8v4
- --spec
- mtp
- --draft-tokens
- "3"

Are you sure 102400 / 409600 is correct for Q6_K? Looks more like 102400 / 204800 . Besides IMO 102400 is quite small for vibe coding.

Progress — weights load fine now, so NInfer-all + this Q6_K pack are OK. The new error is VRAM headroom, not a bad artifact.

requested Engine runtime reservation requires ~13.8 GiB, but only ~9.1 GiB are available means after the 21.8 GiB weights, the engine still wants ~13.8 GiB for the KV/runtime pool and only ~9.1 GiB is left on your RTX 5090. Classic “won’t fit”, not a corrupt file.

On the card numbers: 102400 / 409600 with concurrency 4 is the Q6_K recipe measured on a large card (plenty of leftover VRAM after ~22 GiB weights). On a 32G 5090 that same pool simply doesn’t fit once you also keep --vision + MTP. Your guess of 102400 / 204800 is the right direction for this GPU.

Also fix two command glitches in what you pasted:

1. --max-concurrency. has a trailing dot — that can break parsing. Use --max-concurrency 2 (or 1).
2. You set --kv-dtype twice (fp8 then rk8v4). Keep a single --kv-dtype rk8v4.

Smoke that should fit a 32G card better (drop vision first, then add it back):

ninfer-serve /models/Qwen3.8-27B-Coder390-EfficientThink-Q6_K-MTP.ninfer \
  --host 0.0.0.0 --port 8080 --device 0 \
  --model-id qwen3.8-27b-coder390-q6k-mtp \
  --max-context 102400 --kv-capacity 204800 --kv-dtype rk8v4 \
  --max-concurrency 2 --spec mtp --draft-tokens 3

If that still asks for more runtime than free VRAM, cut again (--max-concurrency 1, and/or --max-context 65536 --kv-capacity 131072), then re-add --vision only after plain text boots.

For vibe coding wanting a longer single context: raise --max-context, but shrink --kv-capacity and concurrency so weights + KV still fit — on 32G you usually trade concurrency for length, you don’t keep the card’s C4 pool.

Fallback: matching GGUF in this repo on upstream llama.cpp with built-in MTP (--spec-type draft-mtp --spec-draft-n-max 4, plus --mmproj if you want vision).

Paste the next startup log if the reservation error persists after the smaller pool.