Instructions to use leoncca/Qwen3.8-27B-Huihui-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use leoncca/Qwen3.8-27B-Huihui-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="leoncca/Qwen3.8-27B-Huihui-AWQ")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("leoncca/Qwen3.8-27B-Huihui-AWQ")
model = AutoModelForMultimodalLM.from_pretrained("leoncca/Qwen3.8-27B-Huihui-AWQ", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use leoncca/Qwen3.8-27B-Huihui-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "leoncca/Qwen3.8-27B-Huihui-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-Huihui-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/leoncca/Qwen3.8-27B-Huihui-AWQ

SGLang

How to use leoncca/Qwen3.8-27B-Huihui-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "leoncca/Qwen3.8-27B-Huihui-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-Huihui-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "leoncca/Qwen3.8-27B-Huihui-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leoncca/Qwen3.8-27B-Huihui-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use leoncca/Qwen3.8-27B-Huihui-AWQ with Docker Model Runner:
```
docker model run hf.co/leoncca/Qwen3.8-27B-Huihui-AWQ
```

Qwen3.8-27B Huihui W4A16 AWQ — 19.55 GB / 18.21 GiB

This is an activation-aware AWQ checkpoint of huihui-ai/Huihui-Qwen3.8-27B-abliterated. The current main branch follows upstream revision 739e3c5b89849f6c238ce1e5b70008612ae42cdd.

The language-model GEMMs are packed as asymmetric W4A16 with group size 128. Vision, native MTP, the LM head, and linear-attention in_proj_a and in_proj_b remain in source precision.

The current revision passed complete artifact audit and native GPTQModel semantic generation on an RTX 3090. It has not yet replaced the live model for an exact 4x V100 runtime smoke. The older V100-validated checkpoint is permanently available at v1-d42ca897.

This is a community quantization, not an official Qwen or Huihui release.

Versions

Version	Mother revision	Validation	Download
`main` / `v2-739e3c5b`	`739e3c5b…`	Artifact audit + RTX 3090 native semantic smoke	files
`v1-d42ca897`	`d42ca897…`	Artifact audit + native smoke + 4x V100 TP4 runtime campaign	files

What changed in the mother checkpoint

Compared with the mother revision used for v1, upstream narrowed the abliteration scope. A tensor-level comparison found 28 matrices restored to official Qwen values in layer indices 15, 16, and 52–63: 14 MLP down_proj, 10 linear-attention out_proj, and 4 self-attention o_proj matrices. The remaining 70 Huihui-modified matrices are in indices 17–51. Vision, MTP, configuration, tokenizer, template, and weight index were unchanged.

Frozen source and quantizer

Component	Repository	Revision
BF16 mother	`huihui-ai/Huihui-Qwen3.8-27B-abliterated`	`739e3c5b89849f6c238ce1e5b70008612ae42cdd`
Quantizer	`modelcloud/GPTQModel` 7.3.2	`f6f1ab12fe8115f84c52253871ca039ac6c425ee`

The pinned mother contains 18 weight shards, 1,199 indexed tensors, and 55,562,855,904 indexed tensor bytes. SOURCE-MANIFEST-BF16-MOTHER.json records the pinned source files and hashes.

Quantization recipe

Item	Value
Method	Activation-aware AWQ, GEMM W4A16
Weight bits	4
Group size	128
Zero point	Asymmetric, enabled
Activation order	Disabled (`desc_act=false`)
Packed dtype	INT32 `qweight` and `qzeros`
Scale dtype	BF16
Quantized modules	400 language-model Linear modules
Calibration	320 frozen records; corpus SHA-256 `de8c3c856c55afd065ade04929659cf990cde7b8cf5fd846c6ee1867b3a13690`
Additional sensitivity exemptions	None
Retained scope	Vision, native MTP, LM head, linear-attention `in_proj_a` and `in_proj_b`

The recipe uses joint in_proj_qkv/in_proj_z AWQ smoothing plus zero-centered RMSNorm-aware BF16 compensation for the retained split-GDN in_proj_a/in_proj_b projections.

Validation of the current version

The v2 release passed these gates:

400/400 target modules packed;
400 qweight, 400 qzeros, and 400 scales tensors stored as INT32, INT32, and BF16;
799 retained source tensors present;
350 structurally protected tensors verified byte-for-byte;
96 retained split-GDN BF16 tensors compensated and audited;
all targets used 202,750 calibration token samples;
native GPTQModel 7.3.2 BACKEND.TORCH FP16 load and semantic smoke passed on an RTX 3090, including exact NATIVE_AWQ_OK output and a natural Chinese response;
every release file is covered by SHA256SUMS.

The current v2 weights do not yet claim exact 1Cat-vLLM/SM70/V100 runtime validation. The v1 tag retains the earlier V100-validated artifact and its historical results.

Download

Current version:

hf download leoncca/Qwen3.8-27B-Huihui-AWQ \
  --revision v2-739e3c5b \
  --local-dir Qwen3.8-27B-Huihui-AWQ-v2

Previous V100-validated version:

hf download leoncca/Qwen3.8-27B-Huihui-AWQ \
  --revision v1-d42ca897 \
  --local-dir Qwen3.8-27B-Huihui-AWQ-v1

Verify either download from inside its model directory:

sha256sum -c SHA256SUMS

Runtime flags depend on the inference engine. Do not assume that a loader which supports older dense AWQ models already supports Qwen3.8 multimodal, linear-attention, native-MTP, and SM70 execution.

License and use

The included model files are released under Apache-2.0, following the source repository. Review the upstream model card and license before use.

The Huihui mother checkpoint intentionally changes refusal behavior. This checkpoint is not safety-tuned, and its outputs may be inaccurate, offensive, or unsafe. Downstream users are responsible for evaluation, safeguards, and legal compliance in their own deployment context.