engharat2/Qwen3.8-27B-QUASAR-NVFP4-NINFER

Qwen3.8-27B QUASAR NVFP4 — NInfer artifact

The QUASAR QAT NVFP4 (W4A4) variant of Qwen3.8-27B, converted into the closed .ninfer artifact format used by the NInfer C++/CUDA inference engine, and evaluated on GPQA-Diamond.

This is a compiled inference artifact, not a transformers-loadable checkpoint. The single file qwen3_8_27b_nvfp4_quasar.ninfer (17.36 GiB) is loaded by ninfer / ninfer-serve, not by transformers. See serving.

Update: I would like to underline that the behavior of the model in actual usage is a bit...weird, with some allucinations happening. As I haven't tested the original QUASAR model, I don't know if the issue comes from my porting, or from QUASAR quantization, or from original Qwen 3.8 NVFP4 model.

Files

File	Size	What
qwen3_8_27b_nvfp4_quasar.ninfer	18,668,301,312 B	the artifact (1,268 objects: 1,262 tensors + 6 frontend resources)
config.json	—	QUASAR model config + base_model lineage + NInfer serving identity
PROVENANCE.json	—	machine-readable pins: sources, fork commit, artifact SHA-256, benchmark

Artifact SHA-256: e75ce1706f4ddd90a4d8a432c3609f23280034f9ff7a804de1e739e948b60c89

Lineage

Qwen/Qwen3.8-27B  (BF16 original)
└─ QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4   (QAT NVFP4 W4A4, vLLM)   ← base_model
   └─ engharat2/Qwen3.8-27B-QUASAR-NVFP4-NINFER   (this repo, NInfer artifact)

Converted with the dual-source converter committed to engharat/ninfer at e80b1ed5 (fork of Neroued/ninfer). Full reproduction steps, design decisions, and per-sample benchmark data live in the documentation repo.

Tensor census

Format	Count	What
NVFP4 (blockscale-k16-m128x4-v1)	256	all 400 fused/standalone text linears' packed weights
BF16 (contiguous-le-v1)	535	norms, token_embedding, GDN conv / a_b, vision, MTP norms
FP32 (contiguous-le-v1)	352	A_log, dt_bias, scales, draft-head ids
FP8_E4M3FN_ROW_BF16S (row-scale-v1)	1	output_head (lm_head)
Q4/Q5/Q6G64_F16S (row-split-k128-v1)	55 / 54 / 1	vision tower + draft head
W8G32_F16S (row-split-k128-v1)	7	5 MTP projections + 2 vision
I32	1	draft-head token ranking

All 400 non-control text linears are bit-identical NVFP4 to the QUASAR source (same packed E2M1 codes, FP8 group scales, and static QAT input_global_scale values). The only deliberate precision changes vs the source: GDN control projections (in_proj_a/in_proj_b, 48 layers) are dequantized to BF16 at conversion (neutral-to-positive), and output_head is row-scaled FP8 (~3% relative logit noise, the sole real compromise).

Benchmark

GPQA-Diamond — the formal NInfer Qwen3.8-27B NVFP4 protocol (198 samples, 0-shot, thinking on, temperature 1.0 / top_p 0.95 / top_k 20, seed 42, concurrency 1, rule-scored). Two independent full runs on this served artifact (MTP spec decoding on, BF16 KV @ 194,560 context):

	Correct	Score
Run 1	179 / 198	0.9040
Run 2	179 / 198	0.9040
Pooled (n=396)	358 / 396	0.9040

Versus the QUASAR model card's vLLM references (same n=396 pooling): BF16 original 0.9141 (362/396), QUASAR via vLLM 0.9091 (360/396), this NInfer artifact 0.9040 (358/396), unsloth NVFP4 0.8939 (354/396), Inferact NVFP4 0.8763 (347/396). The 2-sample gap vs the vLLM QUASAR run is inside the benchmark's sampling noise (binomial σ ≈ 3.7 at n=396, p≈0.91) and tracks the FP8 lm_head.

Serving

It requires a fork of NInfer due to added QUASAR model support and due to custom chat template jinja support.

# NInfer fork at the pinned commit, built:
git clone https://github.com/engharat/ninfer && cd ninfer
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release && cmake --build build --parallel

# Fetch this artifact and serve on an RTX 5090 (32 GB):
hf download engharat2/Qwen3.8-27B-QUASAR-NVFP4-NINFER --local-dir ./out
./build/apps/ninfer-serve out/qwen3_8_27b_nvfp4_quasar.ninfer \
    --host 0.0.0.0 --port 8080 \
    --max-context 194560 --max-concurrency 2 \
    --spec mtp --draft-tokens 3 --lm-head-draft \
    --pending-timeout-ms 300000

Observed: ~5–8k tok/s prefill, ~180–225 tok/s decode (MTP acceptance ~50–70% depending on generation length). --kv-dtype int8 doubles the KV token count at the cost of 8-bit KV.

License

Apache-2.0. Derived from the QUASAR NVFP4 checkpoint (itself derived from Qwen/Qwen3.8-27B); see QUASAR's model card for the upstream licensing terms.