Instructions to use wasifb/Nanbeige4.3-33B-AutoRound-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use wasifb/Nanbeige4.3-33B-AutoRound-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="wasifb/Nanbeige4.3-33B-AutoRound-W4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("wasifb/Nanbeige4.3-33B-AutoRound-W4A16")
model = AutoModelForCausalLM.from_pretrained("wasifb/Nanbeige4.3-33B-AutoRound-W4A16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use wasifb/Nanbeige4.3-33B-AutoRound-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wasifb/Nanbeige4.3-33B-AutoRound-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wasifb/Nanbeige4.3-33B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/wasifb/Nanbeige4.3-33B-AutoRound-W4A16

SGLang

How to use wasifb/Nanbeige4.3-33B-AutoRound-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "wasifb/Nanbeige4.3-33B-AutoRound-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wasifb/Nanbeige4.3-33B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "wasifb/Nanbeige4.3-33B-AutoRound-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wasifb/Nanbeige4.3-33B-AutoRound-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use wasifb/Nanbeige4.3-33B-AutoRound-W4A16 with Docker Model Runner:
```
docker model run hf.co/wasifb/Nanbeige4.3-33B-AutoRound-W4A16
```

Nanbeige4.3-33B — AutoRound INT4 (W4A16)

An INT4 AutoRound quantization of Nanbeige/Nanbeige4.3-33B, a dense 33B reasoning and agentic model (LlamaForCausalLM, 52 layers, GQA 56 / 8 heads, 512K context). It is 20.8 GB and runs on two 24 GB cards with ~200K tokens of fp8 KV cache.

It was made to serve the model on two RTX 3090s in club-3090, with the same AutoRound recipe as our other INT4 builds.

Quantization


Scheme	W4A16: int4, symmetric, group size 128, `auto_round:auto_gptq` packing
Quantized	every linear layer in the 52 decoder layers (364)
Kept in BF16	`lm_head`, embeddings, all norms
Tool	auto-round 0.14.2
Tuning	200 iterations, 512 samples, seqlen 2048, `enable_alg_ext=True`
Calibration	auto-round's default (`NeelNanda/pile-10k`)
Source	`Nanbeige/Nanbeige4.3-33B` @ `c1093c50`

The recipe is also stored in club3090_build_recipe.json. The tokenizer files, chat template and generation_config.json are the original model's, byte for byte.

The original checkpoint has no MTP head, so neither does this one.

Serving

⚠️ Needs transformers 5.x. config.json keeps the RoPE theta (1.2e8) only in rope_parameters, as transformers 5 writes it. A stack on transformers 4.x finds no rope_theta and silently falls back to 10,000, which garbles the output. vLLM v0.32.0 (transformers 5.19) and SGLang v0.5.22 (transformers 5.19) both read it correctly.

vLLM (the AutoRound config resolves to vLLM's inc method on its own):

vllm serve wasifb/Nanbeige4.3-33B-AutoRound-W4A16 \
  --tensor-parallel-size 2 --max-model-len 204800 --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

SGLang:

python -m sglang.launch_server --model-path wasifb/Nanbeige4.3-33B-AutoRound-W4A16 \
  --tp-size 2 --quantization auto-round --context-length 204800 --kv-cache-dtype fp8_e4m3 \
  --chunked-prefill-size 2048 --reasoning-parser qwen3 --tool-call-parser qwen3_coder

On 24 GB cards keep SGLang's --chunked-prefill-size at 2048: 8192 ran out of memory while capturing the prefill CUDA graphs.

The parsers are the ones the original card uses. Its tool format defaults to XML (tool_call_format="xml", parsed by qwen3_coder); json works with hermes. Thinking is on by default (enable_thinking), and preserve_thinking is left to the template, which keeps earlier reasoning: the original card recommends that for agents and false for chat. Sampling follows the original card: agentic and tool use temperature=1.0, top_p=0.95, reasoning and chat 0.6 / 0.95.

Validation

Tested on 2× RTX 3090 (PCIe), tensor parallel 2, fp8 KV, 204,800 context:

	vLLM v0.32.0	SGLang v0.5.22
Loads (Marlin int4 kernels)	yes, 9.83 GiB a card	yes, 10.11 GB a card
fp8 KV pool	207,616 tokens	206,433 tokens
verify-full (answers, tool calls, streaming tool calls, reasoning split, a long answer without repetition)	pass	pass

These are serving checks. Throughput figures and the quality evaluation (club-3090's 8-pack) are pending, and this card will be updated with them.

License

Nanbeige4.3-33B is licensed under Apache-2.0 (see LICENSE). This quantization is a derivative and is distributed under the same license.

All credit for the model goes to the Nanbeige team. This repository only adds the quantization.

Downloads last month: 194

Safetensors

Model size

2B params

Tensor type

I32

BF16

F16

Model tree for wasifb/Nanbeige4.3-33B-AutoRound-W4A16

Base model

Nanbeige/Nanbeige4.3-33B-Base

Finetuned

Nanbeige/Nanbeige4.3-33B

Quantized

(100)

this model