Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,107 @@
+---
+license: apache-2.0 
+base_model:
+- thinkingmachines/Inkling
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- amd-quark
+- mxfp4
+- rocm
+- tokenspeed
+---
+
+# Inkling-MXFP4
+
+## Model Overview
+
+- **Model Architecture:** Thinking Machines Lab Inkling
+  - **Input:** Text, Image, Audio
+  - **Output:** Text
+- **Inference Engine:** [TokenSpeed](https://github.com/lightseekorg/tokenspeed)
+- **Model Optimizer:** [AMD Quark](https://github.com/amd/quark) (0.12.post1+rocm72.torch2.11)
+  - **Quantized layers:** MoE routed experts only
+  - **Weight quantization:** OCP MXFP4, static
+  - **Activation quantization:** OCP MXFP4, dynamic
+
+This model was built by applying AMD Quark MXFP4 quantization to the BF16 Thinking Machines Lab Inkling checkpoint. The quantization targets the MoE routed experts, while attention layers and shared experts are kept in BF16.
+
+## Environment
+
+The quantization workflow was prepared on an AMD gfx950 system. The inspected container environment was:
+
+- **GPU:** AMD MI350/MI355
+- **Target graphics version:** gfx950
+- **ROCm:** 7.2.1
+- **amdgpu driver:** 6.16.13
+- **OS:** Linux 6.8.0-84, x86_64
+- **Python:** 3.12.3
+- **PyTorch:** 2.13.0+rocm7.1
+- **AMD Quark:** 0.12.post1+rocm72.torch2.11
+- **Safetensors:** 0.8.0
+- **Transformers:** 5.13.1
+
+Create and activate the Quark environment:
+
+```bash
+python3 -m venv ~/.venv-quark
+source ~/.venv-quark/bin/activate
+```
+
+Install the required packages:
+
+```bash
+python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm7.1
+python -m pip install amd-quark --extra-index-url https://pypi.amd.com/quark/rocm72/simple
+python -m pip install safetensors transformers accelerate tqdm
+```
+
+## Model Quantization
+
+The model was quantized with the Quark file-to-file flow. This avoids loading the full BF16 checkpoint into GPU memory at once, which is important for very large MoE checkpoints. Run the quantization script:
+
+```bash
+python quantize_quark.py \
+  --model_dir /path/to/model \
+  --output_dir /path/to/output \
+  --quant_scheme mxfp4 \
+  --file2file_quantization
+```
+
+The script applies the model-specific exclusion policy automatically in file-to-file mode. The resulting checkpoint stores MXFP4 routed-expert weights and scales while preserving non-routed-expert components in BF16.
+
+## Deployment
+
+This model can be served with [TokenSpeed](https://github.com/lightseekorg/tokenspeed):
+
+```bash
+tokenspeed serve \
+  --model lightseekorg/Inkling-MXFP4 \
+  --attn-tp-size 4 \
+  --moe-tp-size 4 \
+  --max-model-len 81920 \
+  --max-num-seqs 16 \
+  --max-prefill-tokens 8192 \
+  --chunked-prefill-size 8192 \
+  --gpu-memory-utilization 0.95 \
+  --disable-cuda-graph-padding \
+  --trust-remote-code \
+  --dtype bfloat16 \
+  --disable-kvstore \
+  --kvstore-ratio 0 \
+  --block-size 128 \
+  --host 127.0.0.1 \
+  --port 22015
+```
+
+## Evaluation
+
+The following validation results are placeholders and will be updated before public release.
+
+| Benchmark | BF16 Reference | MXFP4 |
+| --- | ---: | ---: |
+| BFCL exact calls | 78.3% | 79.1% |
+| BFCL all-live macro | 75.4% | 75.3% |
+| MMAU | 77.2% | 76.0% |
+| GPQA Diamond | 88.1% | 85.4% |
+| AIME26 | 96.4% | 96.7% |