Instructions to use btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16")
model = AutoModelForMultimodalLM.from_pretrained("btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16

SGLang

How to use btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16 with Docker Model Runner:
```
docker model run hf.co/btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16
```

Qwen3.8-Flash-Next NVFP4 (LIL) with BF16 PLE tables

A derivative of local-inference-lab/Qwen3.8-Flash-Next-NVFP4 at revision 6c01ac378d22f6652c9f392c87700496eca8e5dc, itself a quantization-aware-distilled build of Qwen/Qwen3.8-Flash-Next. Public, ungated. No retraining was done.

What changed

PLE tables = LIL's QAD-trained NVFP4 tables dequantized to BF16 (no retraining, no other weights changed).

The 128 n-gram (PLE) tables model.language_model.layers.1.ple.ple_embedding.ngram_embedding.shard_{0..127}.weight were dequantized from NVFP4 (E2M1 values x E4M3 block scale per 16 elements x FP32 global weight_scale_2 = 3.3242362e-05) in FP32 and rounded to BF16 once. They are stored as BF16 [2500012, 160] in ple-bf16-00..42.safetensors (3 tables per file, 102.4 GB total). The .weight_scale tensors and weight_scale_2 are removed.
config.json: text_config.ple_embedding_dtype nvfp4 -> bfloat16; the PLE entry removed from quantization_config.quantized_layers. hf_quant_config.json: the same entry removed.
model.safetensors.index.json regenerated for the new layout. model-00026-of-00041.safetensors was rewritten without the PLE tensors (its three small ple.norm_* tensors are copied byte-for-byte); model-00027..00041 were dropped because they contained only PLE tensors.
Every other tensor and file is byte-identical to the source revision, including attention and shared-expert MXFP8, routed-expert NVFP4 and the MTP layer.

Why: runtimes that load the PLE table to host memory only for BF16 (for example vLLM's PLE CPU-offload path) cannot load NVFP4 PLE tables.

Rounding footnote: BF16 rounding of the FP32 dequant has max relative error 0.00307 and mean 0.0013 on sampled rows (bound 2^-8 = 0.0039). Against the original unquantized BF16 PLE table, a 400-row sample of shards 0-2 gives cosine ~0.995 and relative L2 ~0.097; that difference is the NVFP4 quantization the LIL model was trained with, not a conversion error. This repo does not use the original table.

Not validated end to end here: no perplexity or generation test of this exact checkpoint has been run by the publisher of this derivative. Note that export-manifest.json and HYBRID.json are carried over from the source unchanged and still describe the source's NVFP4 PLE.

Serving note (vLLM)

With vLLM 0.31's quantization resolver (checked on CPU against this repo's config.json / hf_quant_config.json, with LIL @6c01ac37 as the control), the PLE table resolves to the unquantized embedding method, i.e. BF16 tables that can be held in pinned host memory with the PLE CPU-offload mode (--engram-config cpu_offload); the MTP draft experts (mtp.layers.*.mlp.experts) resolve to W4A16_NVFP4 and the routed experts to NVFP4, unchanged from the source. Expect about 95 GB of host RAM for the BF16 tables. This resolver check is not a full serving benchmark; no quality or throughput numbers are claimed for this exact checkpoint.

Provenance and license

Source: local-inference-lab/Qwen3.8-Flash-Next-NVFP4 @ 6c01ac378d22f6652c9f392c87700496eca8e5dc (all LFS files sha256-checked against the Hub before use).
Upstream base: Qwen/Qwen3.8-Flash-Next. License: Qwen Community License 1.0, carried over (LICENSE). See the source card for training details and its own terms.
Built and uploaded by btbtyler09.

Downloads last month: 72

Safetensors

Model size

118B params

Tensor type

I64

BF16

F32

F8_E4M3

Model tree for btbtyler09/Qwen3.8-Flash-Next-NVFP4-LIL-PLEbf16

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

local-inference-lab/Qwen3.8-Flash-Next-NVFP4

Quantized

(5)

this model