Instructions to use Qwen/Qwen3-Next-80B-A3B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Qwen/Qwen3-Next-80B-A3B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Qwen/Qwen3-Next-80B-A3B-Instruct") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct") model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-Next-80B-A3B-Instruct", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Inference
- HuggingChat
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Qwen/Qwen3-Next-80B-A3B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Qwen/Qwen3-Next-80B-A3B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Qwen/Qwen3-Next-80B-A3B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Qwen/Qwen3-Next-80B-A3B-Instruct
- SGLang
How to use Qwen/Qwen3-Next-80B-A3B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Qwen/Qwen3-Next-80B-A3B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Qwen/Qwen3-Next-80B-A3B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Qwen/Qwen3-Next-80B-A3B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Qwen/Qwen3-Next-80B-A3B-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Qwen/Qwen3-Next-80B-A3B-Instruct with Docker Model Runner:
docker model run hf.co/Qwen/Qwen3-Next-80B-A3B-Instruct
FP8 please
Any chances for FP8 variant, like you did for Qwen3-30B-A3B models? Will be much appreciated.
Super!
Is there any receipt to produce correct FP8 quant for qwen3-next preserving MTP Draft model? llmcompressor? It looks like FP8 Qwen3-Next-80B-A3B produces no Accepted Tokens.
[metrics.py:96] SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00 tokens/s, Drafted throughput: 69.20 tokens/s, Accepted: 0 tokens, Drafted: 692 tokens, Per-position acceptance rate: 0.000, 0.000, Avg Draft acceptance rate: 0.0%
I use this model: https://huggingface.co/DevQuasar/Qwen.Qwen3-Next-80B-A3B-Instruct-FP8-Dynamic
Is it possible to support correct FP8 quantization of the Draft model for spec decoding? Any receipt for llmcompressor is highly apppreciated!
Currently qwen3-next fp8 support is not good, lots of bugs.
maybe we should wait for the official fp8 release.
Unfortunately doesn't work with vLLM when offloading to CPU is enabled.
RuntimeError: Worker failed with error 'Cannot re-initialize the input batch when CPU weight offloading is enabled. See https://github.com/vllm-project/vllm/pull/18298 for more details.'
OK, will wait for llama.cpp to develop Qwen-Next support and then for quantized GGUFs to become available.
In any case, thanks to Qwen team for their models!
vllm v0.10.2 does not support qwen3 next fp8
maybe latest can run
I ran it with the latest (pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly) according to the recommendations in the model description. vllm v0.10.2 indeed fails earlier because of quantization.
