--- license: apache-2.0 language: - pl - en base_model: speakleash/Bielik-4.5B-v3.0-Instruct library_name: transformers pipeline_tag: text-generation tags: - typed-decisions - decision-making - classification - calibration - polish - fp8 - decision-model --- # basal-2.0-4.5B-FP8 [![GitHub](https://img.shields.io/badge/GitHub-rkinas%2Fbasal%20%E2%80%94%20inference%20engine-black?logo=github)](https://github.com/rkinas/basal) [![Website](https://img.shields.io/badge/web-basal.si5.pl-0a7d5a)](https://basal.si5.pl/) [![Collection](https://img.shields.io/badge/%F0%9F%A4%97-basal--2.0%20collection-yellow)](https://huggingface.co/collections/Remek/basal-20-6acd1712a8e443c86cd576fc) FP8 checkpoint for NVIDIA Hopper and Blackwell GPUs of [basal-2.0-4.5B](https://huggingface.co/Remek/basal-2.0-4.5B) (4.5B, base Bielik-4.5B-v3.0-Instruct). Same model, same typed-decision API through `basal-serve`; the calibration files this folder ships are listed below. > Passes every bar of this tier against bf16 (agreement, agreement near ties, accuracy, ECE). ## Measured against bf16 **Agreement** is the share of decisions where the port picks the same option as the bf16 release model on the development set dev v2 (MLX rows: every 3rd dev v2 decision, 1,700 decisions). Bars for this tier: agreement 2-10 >= 0.9415, agreement at reference margin >= 0.2 >= 0.9863, accuracy within 1 point of bf16, ECE at most +0.01. All ports of this tier: | package | agreement with bf16 (options 2-10) | agreement at margin >= 0.2 | accuracy diff | ECE diff | size | speed | gate | |---|---|---|---|---|---|---|---| | [FP8](https://huggingface.co/Remek/basal-2.0-4.5B-FP8) | 0.9478 | 0.9890 | -0.0008 | +0.0009 | 4.9 GB | 38.6 dec/s c16, c1 p50 29 ms (H100) | PASS | | [GGUF Q8_0](https://huggingface.co/Remek/basal-2.0-4.5B-GGUF) | 0.9839 | 0.9988 | +0.0002 | +0.0035 | 4.8 GB | not measured in this release | PASS | | [GGUF Q5_K_M](https://huggingface.co/Remek/basal-2.0-4.5B-GGUF) | 0.9569 | 0.9920 | -0.0039 | +0.0040 | 3.2 GB | not measured in this release | PASS | | [MLX 8-bit](https://huggingface.co/Remek/basal-2.0-4.5B-MLX-8bit) | 0.9792 | 0.9989 | +0.0000 | -0.0009 | 4.9 GB | 278 ms c1 p50, 698 ms for 5 questions (Mac) | PASS | The ports are decision-layer checks against bf16, not new benchmarks; the model's own results are on [basal-2.0-4.5B](https://huggingface.co/Remek/basal-2.0-4.5B). ## Use The same install as the bf16 model ([quick start](https://huggingface.co/Remek/basal-2.0-4.5B#quick-start)); then: ```bash basal-serve --model Remek/basal-2.0-4.5B-FP8 --mode fp8 --template-prefix off --port 8000 ``` `--template-prefix off` is needed for this model: the prompt-template cache failed its check. An NVIDIA ModelOpt FP8 checkpoint that the basal engine v2.0.0 loads natively; needs a GPU with FP8 (sm89+, Hopper or Blackwell). In FP8 the batch can change near-tie decisions; `--deterministic` runs every request alone with full FP8 accumulation. Speed: 38.6 dec/s c16, c1 p50 29 ms (H100); one NVIDIA H100 80GB (cluster node), one option order. **Engine.** FlashAttention 2; compact rows and template prefix off (FP8 keeps FlashAttention 2 unless `--attn fa3`, which is faster but was not gated and changes some decisions near ties). Engine v2.0.0 schedules long requests (over 2,048 tokens) behind short ones: in a mixed load at 32 clients short requests answer in 0.81 s instead of 1.38 s (p50), and long requests take about 7% longer (p50 1.96 s instead of 1.83 s); `--scheduler off` serves in arrival order. The same basal engine v2.0.0 also serves the experimental basal-omni-1.0 models (images and audio): [README](https://github.com/rkinas/basal#images-and-audio-basal-omni-experimental). ## Calibration `CALIBRATION.json` in this folder is a copy of the FP8 fit (`CALIBRATION.fp8.json`). basal-serve `--mode fp8` loads `CALIBRATION..json` for its engine when the folder has it, else `CALIBRATION.json`. A calibration fitted on bf16 readouts does not carry over exactly to a quantised port (an outside test of basal-1.5 on Ollama saw 2.3% real error at the 1% threshold with the bf16 file), so a port-specific fit, where it passed, uses the bf16 model's method on that port's own readouts of the same calibration splits, one option order: - `basal-serve --mode fp8`: Ships `CALIBRATION.fp8.json`, fitted on FP8 readouts of this port (same method as the bf16 fit, one option order); dev v2 ECE 0.0184 with the bf16 fit -> 0.0215 with this fit (in-sample), 0.0241 held-out (leave-one-family-out). Where a port uses the bf16 calibration, treat its confidence thresholds as approximate and fit your own with `basal-calibrate` when they matter. ## Results of the model See the model card [basal-2.0-4.5B](https://huggingface.co/Remek/basal-2.0-4.5B) for Werdykt, JDI, LEK, PL-Holdout and PL-Real numbers against basal-1.5 and the evaluation notes. This port is not scored separately on those benchmarks. ## Limitations - Validate on your own documents before relying on it: hidden and held-out sets do not cover every domain. - Polish world knowledge of a small model is limited: put the relevant facts in the state. Legal rules change. - Exact arithmetic and thresholds are a weak spot of one-pass decisions: compute them in code (or `"facts": "auto"`). - The calibration is cautious on unseen tasks: no 1% threshold is certified in the shipped files, so `"max_error": 0.01` on an `act` or `tool` question defers every decision; use 0.05, or certify thresholds on your own data with `basal-calibrate`. - `act` and `multi` over many labels are experimental. Up to 255 options per question (Ollama: 20); the context is 32,768 tokens. The 2.0 models ship no evidence head. - Decisions with serious consequences for people should be reviewed by a person. ## Citation ```bibtex @software{kinas2026basal20, title = {basal-2.0: typed-decision models and inference engine for Polish and English}, author = {Kinas, Remigiusz}, year = {2026}, version = {2.0.0}, url = {https://github.com/rkinas/basal} } ``` ## License and attribution Apache-2.0. Fine-tuned from [speakleash/Bielik-4.5B-v3.0-Instruct](https://huggingface.co/speakleash/Bielik-4.5B-v3.0-Instruct) (Apache-2.0). Bielik is a project of SpeakLeash and ACK Cyfronet AGH.