Instructions to use bowmanslayer/VeriLoop-E2-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bowmanslayer/VeriLoop-E2-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bowmanslayer/VeriLoop-E2-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("bowmanslayer/VeriLoop-E2-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("bowmanslayer/VeriLoop-E2-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bowmanslayer/VeriLoop-E2-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bowmanslayer/VeriLoop-E2-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bowmanslayer/VeriLoop-E2-W4A16

SGLang

How to use bowmanslayer/VeriLoop-E2-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bowmanslayer/VeriLoop-E2-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bowmanslayer/VeriLoop-E2-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bowmanslayer/VeriLoop-E2-W4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bowmanslayer/VeriLoop-E2-W4A16 with Docker Model Runner:
```
docker model run hf.co/bowmanslayer/VeriLoop-E2-W4A16
```

VeriLoop-E2-W4A16

Community quantization of tsinghua-sigs-robot-lab/VeriLoop-E2, fixed revision 9379d199adcc44acdf836558037163878e3a37ae. This is not an official Tsinghua release. Model architecture is Qwen3_5ForConditionalGeneration; the upstream names the base Qwen3.8-27B.

What is included

AutoRound W4A16, symmetric group 128, 256 public Pile calibration samples, 512 tokens, 200 iterations, batch 4. Eligible weights are INT4; small hybrid projections, activations and retained weights are BF16. All source tensors are represented; the 333 vision tensors are preserved exactly.

No abliteration was applied to this version.

Local paired evaluation

Same frozen 1,514 questions, seed 20260816, thinking enabled with the original template and publisher raw-content protocol, greedy decoding, max output 32,768 tokens. These are our legacy sampled/custom scorers, not official leaderboard results. IFEval uses a partial custom rule implementation. See EVALUATION.md and evaluation.json for details and validity flags.

Task	n	Original w4a16	Ablated w4a16	Delta pp
mmlu	150	84.00%	76.00%	-8.00
cmmlu	150	84.67%	75.33%	-9.33
mmlu_pro	150	78.67%	70.67%	-8.00
ceval	150	84.00%	76.00%	-8.00
arc	150	80.67%	85.33%	+4.67
truthfulqa	150	58.67%	71.33%	+12.67
gsm8k	100	95.00%	88.00%	-7.00
math500	100	75.00%	73.00%	-2.00
bbh	150	75.33%	65.33%	-10.00
humaneval	164	89.63%	89.63%	+0.00
ifeval	100	76.00%	69.00%	-7.00

Overall original → ablated: 80.05% → 76.42%, difference -3.63 pp; approximate paired 95% CI [-5.894, -1.371] pp. Empty responses: [9, 8]; length-limited responses: [9, 8]. Both stay in the denominator. Empty-rate validity gate: False.

On 100 separate refusal probes (thinking disabled, 2048-token budget), opening-30-word marker counts were 98/100 → 0/100. Marker absence is not proof of semantic compliance. Truncation/budget hits: [1, 9]; validity: [True, True]. The first 16 probes overlap candidate selection; see the 84-item held-out subset in evaluation.json when available.

Loading

vllm serve /path/to/VeriLoop-E2-W4A16 --tensor-parallel-size 2 --generation-config vllm

The measured runtime was vLLM 0.29.0 with a local FA2/FP8-KV plugin on dual RTX 3090. That local setup requires its plugin-specific library environment variables; this repository does not bundle the plugin. The command above is a starting point for a compatible installed runtime, not a tested stock-vLLM support matrix. No remote custom model code is required. Vision weights are retained exactly, but visual inference is not validated in this evaluation.

Scope and limitations

Use the repository-shipped template and publisher raw-content serving protocol. Do not add a Qwen3 reasoning parser: some completed direct answers omit a closing think tag and that parser classifies them entirely as hidden reasoning. Removing it recovered all 33 initial missing-answer cases; streaming/nonstreaming checks passed. Thinking remains enabled. For evaluation, final content is taken after </think> when present; normally stopped direct content is retained, while unclosed length-limited output is not scored as an answer.

In llama.cpp content-only mode, remove an echoed opening <think> generation-prefix tag before applying answer extraction. It is protocol framing, not answer text. The GGUF preflight was restarted after verifying this correction and the template override; it is excluded from the reported scores.

The author's private VeriLoop Harness is not included or reproduced. Their SWE/Terminal/DeepSWE scores are not scores for these weights. This evaluation does not establish long-context, tool-use, vision, agentic performance, or production suitability. Reduced refusal can affect inappropriate-content handling; applications need their own behavior controls. Full semantic refusal review and comprehensive capability equivalence have not been established.

Apache-2.0 is retained from the fixed upstream source; see LICENSE. Source training belongs to its original authors. Community changes are quantization. See release.json and SHA256SUMS for provenance and integrity.