Instructions to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS
# Run inference directly in the terminal:
./llama-cli -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Use Docker

docker model run hf.co/julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

LM Studio
Jan

vLLM

How to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Ollama
How to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with Ollama:
```
ollama run hf.co/julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS
```
Unsloth Desktop

How to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with Docker Model Runner:
```
docker model run hf.co/julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS
```

Lemonade

How to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Run and chat with the model

lemonade run user.Qwen3.8-Flash-Next-IQ4_XS-GGUF-IQ4_XS

List all available models

lemonade list

Hermes Agent

How to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF:IQ4_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-Flash-Next GGUFs — provenance-verified quants for Strix Halo

Four files:

file	size	what it is
`Qwen3.8-Flash-Next-IQ4_XS-PLE.gguf`	91 GiB	recommended daily driver — iq4_xs trunk with the 27G PLE n-gram table at iq4_nl
`Qwen3.8-Flash-Next-IQ4_XS.gguf`	116 GiB	static reference quant, PLE at q8_0
`Qwen3.8-Flash-Next-IQ4_XS-M2.gguf`	115 GiB	imatrix-calibrated quant, PLE table at q8_0 — best measured perplexity
`mtp-Qwen3.8-Flash-Next-Q8_0.gguf`	3.9 GiB	MTP draft sidecar for nathanw1014-lineage engines (fork-specific — will NOT load on apepojken/mainline)

M2 — the imatrix quant

second-generation quant: same trunk type (iq4_xs), but calibrated with a 926-entry imatrix (1,024 chunks) via the ROCmFPX banded quantizer, and the 51B PLE lookup table left at q8_0 (no --tensor-type cut). 5.56 bpw, 115 giB — 24 giB bigger than the 91g PLE file.

perplexity (wiki.test.raw, ctx 2048, 145 chunks):

quant	PPL
M2 (imatrix, PLE q8_0)	4.2809 ±0.025
PLE 91g	4.2932 ±0.025 (statistically tied, <0.5σ)
static 116g	4.5221 ±0.026 (~9σ worse)

speed profile is honest-mixed (single runs, nathanw1014 vulkan engine, q8_0 kv, temp 0):

depth	plain tg	mtp tg
8k	23.8 (≈ PLE 23.8)	25.1 (PLE 33.5 — M2 slower)
32k	19.0 (≈ PLE 19.0)	29.1 (best of the three)
128k	—	13.3 (PLE 13.5 — tied)

pick M2 when you want the best measured quality and don't mind the 24 giB: at ≤32k plain it matches PLE, and at 32k MTP it measured fastest. for shallow-depth MTP speed take the 91g PLE; for deep 128k+ MTP the static 116g had a small in-sweep edge (17.4 vs 13.3/13.5 — within the same-config spread, n=1 caveat).

provenance

every quant descends from an F16 that was byte-verified against the official Qwen/Qwen3.8-Flash-Next-FP8 checkpoint: hyper-connection norms folded to (1 + w) (97/97 tensors — the converter bug that produces deterministic garbage is fixed in our pipeline), PLE fp8 scale applied, expert stacking identity probed 512x3, GDN v-head reorder checked. details: https://github.com/julianmb/haloq38flash

the PLE cut (what makes the 91G special)

the 51B-parameter n-gram lookup table was moved from q8_0 (54G) to iq4_nl (27G) via --tensor-type. hash-gathered lookup rows tolerate the precision drop — verified by smoke and full benchmark, no degradation observed:

depth	static 116G plain/mtp t/s	PLE 91G plain/mtp t/s
0	29.2 / 48.4	29.9 / 53.1
8k	22.9 / 42.8	24.1 / 56.4
32k	19.5 / 29.5	20.1 / 30.2

prefill at 32k: 384 → 397 t/s. no collapse at depth. engine: nathanw1014 strix-halo-vulkan (ad914eb), vulkan/radv, q8_0 KV, -ub 2048, temp 0.

fork compatibility caveat (important)

the iq4_nl PLE rows assert in SOME forks: engines that feed gathered PLE rows directly as mul_mat B operands without dequantizing (apepojken qwen4exp-spec-mtp) abort at ggml-vulkan.cpp:7794 (b_type must be F32/F16/Q8_1). verified working on nathanw1014 strix-halo-vulkan. if your engine asserts on load or first token, use the 116G static file instead. M2 keeps the PLE table at q8_0 and has no such assert exposure.

provenance note

the same --tensor-type cut applied to unverified-source quants will NOT fix a broken converter (hc norms, PLE scale) — garbage in, garbage out. ours is built from a fixed, audited pipeline.

128k & 256k context benchmarks (SSD-PLE)

the depth story is not monotonic. measured on the same engine (nathanw1014 vulkan, q8_0 kv, temp 0):

depth	static 116g plain/mtp t/s	PLE 91g plain/mtp t/s
0	29.2 / 48.4	29.1 / 48.1
8k	22.9 / 42.8	21.8 / 27.9 (peak: 56.4)
32k	19.5 / 29.5	18.5 / 25.5 (peak: 30.2)
128k	10.8 / 26.9	8.9 / 11.8 (peak: 18.6)
256k	6.2 / —	6.0 / 15.2 (SSD-PLE tuned)

256k context unlocked with SSD-PLE: With -lm mmap --tensor-read-lazy on, the 27 GB PLE table is kept on NVMe SSD, keeping active RAM under 95 GB and providing ~30 GB of free headroom at full 256k context. Combined with -tb 16 and -ub 1024 -b 2048 -t 4, this achieves 15.2 t/s MTP generation and 179-191.5 t/s prefill at 256k with zero swap thrashing.
Hardware boundary: Do not use -ub 2048 at 256k context; it triggers Vulkan queue submission timeouts (ErrorDeviceLost). -ub 1024 is the tested maximum.
128k reversal: At 128k specifically under MTP, the static 116g quant has higher draft acceptance than the 91g PLE quant (26.9 vs 18.6 t/s).
Practical: Use the 91g PLE file for daily chat, coding, and full 256k long context (with SSD-PLE flags). Use the 116g static file if you specifically need highest MTP speed at 128k.

run it (strix halo, 128 GB unified memory)

full methodology, receipts, and the benchmark record: https://github.com/julianmb/haloq38flash

docker (one-liner; image default serves the 91g PLE on :8080):

git clone https://github.com/julianmb/haloq38flash && cd haloq38flash
docker compose up --build

point it at M2 with the MTP sidecar and the warm-turn cache:

docker compose run qwen38-flash-next /app/llama-server \
  -m /models/Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
  -md /models/mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  --cache-ram 8192 --ctx-checkpoints 32 \
  -c 32768 -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4

or raw llama.cpp (nathanw1014 strix-halo-vulkan lineage engines):

llama-server -m Qwen3.8-Flash-Next-IQ4_XS-M2.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 6 --spec-draft-p-min 0.75 \
  -ngl 999 -fa on -ctk q8_0 -ctv q8_0 -ub 2048 -t 4 -c 32768

perf notes: -t 16 lifts prefill up to +43% at 128k (decode indifferent); the --cache-ram/--ctx-checkpoints warm-turn flags make repeated context nearly free (measured 438 s → 0.68 s at 128k, 994 s → 0.74 s at 256k — 640x/1351x). swap M2's filename for the other quants; same flags.

license

qwen community license 1.0 (distribution permitted with notice; maas restrictions apply). base model: Qwen/Qwen3.8-Flash-Next.

Downloads last month: 18,954

GGUF

Model size

177B params

Architecture

qwen4exp

Hardware compatibility

4-bit

8-bit

Model tree for julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

Qwen/Qwen3.8-Flash-Next-FP8

Quantized

(14)

this model

julianmb
/

Qwen3.8-Flash-Next-IQ4_XS-GGUF

Qwen3.8-Flash-Next GGUFs — provenance-verified quants for Strix Halo

M2 — the imatrix quant

provenance

the PLE cut (what makes the 91G special)

fork compatibility caveat (important)

provenance note

128k & 256k context benchmarks (SSD-PLE)

run it (strix halo, 128 GB unified memory)

license

Model tree for julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF

Space using julianmb/Qwen3.8-Flash-Next-IQ4_XS-GGUF 1