---
license: mit
license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B/blob/main/LICENSE
base_model: deepreinforce-ai/Ornith-1.0-9B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
tags:
  - gguf
  - autoround
  - intel-autoround
  - quantized
  - llama.cpp
  - code
  - agent
---

# Ornith-1.0-9B AutoRound GGUF

GGUF quantizations of [**deepreinforce-ai/Ornith-1.0-9B**](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B),
produced with [Intel **AutoRound**](https://github.com/intel/auto-round) using its `alg_ext`
iterative calibration (~200 optimization steps per block) rather than plain round-to-nearest.
At a given size this recovers noticeably more quality than vanilla GGUF quants, with the largest
benefit at the low bit widths.

Ornith-1.0 is a self-improving, MIT-licensed family of open-source **agentic-coding** models from
Deep Reinforce; the 9B is the lightweight, single-GPU member (dense, 32 layers, 256K context). See
the [original model card](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B) for capabilities
and coding benchmarks. This repo only re-packages those weights as GGUF.

## Quality vs. size

Every quant was measured against the **BF16** reference on **wikitext-2 test**: PPL and, more
importantly, **KL-divergence** of the output distribution (how far the quant drifts from full
precision; lower is better). **Top-1** is the share of positions whose argmax token still matches
BF16. Reference BF16 PPL = **8.13**.

![Quality vs. size on wikitext-2: Mean KLD (top) and PPL (bottom) against bits-per-weight](pareto.png)

| Scheme | Size | bpw | PPL | ΔPPL | Mean KLD | Top-1 agree |
|--------|-----:|----:|----:|-----:|---------:|------------:|
| Q2_K_S | 3.7 GB | 3.30 | 9.89 | +21.7% | 0.2960 | 76.8% |
| Q3_K_S | 4.3 GB | 3.81 | 8.53 | +4.9% | 0.1216 | 85.0% |
| Q3_K_M | 4.5 GB | 4.04 | 8.57 | +5.4% | 0.1214 | 84.9% |
| Q3_K_L | 4.9 GB | 4.40 | 8.60 | +5.8% | 0.1201 | 84.9% |
| **Q4_K_S** | 5.4 GB | 4.78 | 8.05 | -1.0% | 0.0425 | 91.0% |
| **Q4_K_M** ⭐ | 5.6 GB | 5.03 | 8.08 | -0.6% | 0.0406 | 91.2% |
| Q6_K | 7.4 GB | 6.58 | 8.15 | +0.3% | 0.0040 | 96.6% |
| Q8_0 | 9.5 GB | 8.51 | 8.14 | +0.1% | 0.0020 | 97.5% |

**Recommendations**
- **Q4_K_M** ⭐ is the best all-round pick: smaller than the Q5 tier yet closest of the mid-size
  quants to BF16 (KLD 0.04, Top-1 91%). Start here.
- **Q6_K / Q8_0** are effectively lossless (KLD under 0.004) when you have the VRAM.
- **Q3_K_*** are usable on tight memory. Reach for **Q2_K_S** only when you must, since 2-bit shows
  noticeable drift.

**Why no Q5 or Q2_K_MIXED?** On the KLD benchmark the Q5_K quants were *not* closer to BF16 than
Q4_K_M despite being larger, so they add size without quality for this model and were omitted. The
`Q2_K_MIXED` variant was defective here (broken output distribution) and was also excluded. (This
base checkpoint ships no MTP/NextN head, so there is no speculative-decoding block to preserve.)

## Multimodal

Ornith inherits a vision tower. Multimodal projectors are included for use via `llama-mtmd-cli`
(pair any LM quant above with one of these). For text-only / coding use you can ignore them.

| Projector | Size | Notes |
|-----------|-----:|-------|
| `mmproj-Ornith-1.0-9B-f16.gguf` | 0.9 GB | recommended (half size, no visible quality loss) |
| `mmproj-Ornith-1.0-9B-bf16.gguf` | 0.9 GB | bf16 alternative |
| `mmproj-Ornith-1.0-9B-f32.gguf` | 1.8 GB | full precision |

## Usage (llama.cpp)

```bash
# chat / completion (the model's chat template, incl. <think> reasoning, is embedded)
llama-cli   -m Ornith-1.0-9B-Q4_K_M.gguf -ngl 999 -c 8192 -p "Write a Python LRU cache."

# OpenAI-compatible server
llama-server -m Ornith-1.0-9B-Q4_K_M.gguf -ngl 999 -c 8192

# multimodal (image + text)
llama-mtmd-cli -m Ornith-1.0-9B-Q4_K_M.gguf \
               --mmproj mmproj-Ornith-1.0-9B-f32.gguf \
               --image picture.jpg -p "Describe this screenshot."
```

## How these were made

```bash
auto-round --model deepreinforce-ai/Ornith-1.0-9B \
           --scheme gguf:q4_k_m --enable_alg_ext --enable_torch_compile \
           --output_dir ./quantized
```

All outputs were NaN-scanned and inference-checked. The comparison table and the Pareto plot above
(`pareto.png`, `comparison.csv` in this repo) come from `llama-perplexity --kl-divergence` against
the BF16 reference.

## Support

These quantized models are made in my spare time using expensive hardware such as DGX Spark systems for quantization and validation. If you find these GGUFs useful for your projects, consider buying me a coffee to help cover hardware and compute costs. Every bit of support helps me keep producing high-quality quantized models for the community!

[☕ Support me on Ko-fi](https://ko-fi.com/sphaela)

## Credits & license

- Base model © Deep Reinforce, **MIT** ([LICENSE](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B/blob/main/LICENSE)).
- Quantization: [Intel AutoRound](https://github.com/intel/auto-round) (Apache-2.0) and [llama.cpp](https://github.com/ggml-org/llama.cpp).