chiruan/DeepSeek-R1-Distill-Qwen-7B-quarot-w4a8kv4-tp1

DeepSeek-R1-Distill-Qwen-7B QuaRot W4A8KV4

QuaRot checkpoint derived from DeepSeek-R1-Distill-Qwen-7B, exported for tensor parallelism TP=1.

Setting	Value
Weight precision	4 bits
Activation precision	8 bits, asymmetric
Key / value cache precision	4 / 4 bits, asymmetric
Key / value group size	128 / 128
Saved tensor dtype	BF16
Architecture	Qwen2QuaRotForCausalLM

Format and loading

This is a fake-quantization checkpoint: the quantized weights are stored as BF16 tensors in four Safetensors shards. The checkpoint contains about 15.24 GB of model and tokenizer files. Activation and KV-cache quantization are applied by the custom runtime according to config.json. The bit widths describe simulated numerical precision, not packed integer storage.

Use the compatible custom vLLM runtime from Quantized-Reasoning-Models, including Qwen2QuaRotForCausalLM and its quantization/Hadamard utilities. Plain Transformers loading does not reproduce this custom quantization behavior. The runtime code is maintained separately from this checkpoint.

After installing that project's dependencies, run from its checkout:

from huggingface_hub import snapshot_download
from vllm import LLM
from vllm_custom.model_executor.fake_quantized_models.registry import register_fake_quantized_models

checkpoint = snapshot_download("chiruan/DeepSeek-R1-Distill-Qwen-7B-quarot-w4a8kv4-tp1")
register_fake_quantized_models()
llm = LLM(
    model=checkpoint,
    dtype="bfloat16",
    tensor_parallel_size=1,
    max_model_len=32768,
    gpu_memory_utilization=0.85,
)

The uploaded weights, configuration, and tokenizer files preserve the local evaluated checkpoint. This card does not assert benchmark scores; evaluation was ongoing at upload time.

License

The base model is distributed under the MIT license. Its license notice is included in LICENSE; see the base model card for upstream provenance.