Comparison tables: BTZSC-22 with DiffusionGemma-26B-A4B add-on (Track A), speed/concurrency/GPU-memory run 3 (Ines-1, laya, GLiNER2.5-multi, Gemma)

README.md

@@ -36,7 +36,7 @@ datasets:
 >   `tokenizer/`; the documentation (`README.md` and the other `.md` files, `paper/`, `assets/`, `tasksource_license_audit.csv`); the
 >   training records (`training/*.json`); the model's own evaluation outputs in `eval/`; `SHA256SUMS`.
 > - **MIT** ([LICENSE-CODE](LICENSE-CODE)): the code — `mini_v41/`, `mini_v41_jev/`, `scripts/`, `examples/`,
->   `training/code/`, `eval/btzsc22/code/` — and the environment and build files: `Dockerfile`, `.dockerignore`, `requirements.txt`,
+>   `training/code/`, `eval/btzsc22/code/`, `eval/btzsc22/addon_gemma/code/` — and the environment and build files: `Dockerfile`, `.dockerignore`, `requirements.txt`,
 >   `requirements.lock`, `.gitattributes`, `.gitignore`.
 > - **Third-party terms, not relicensed:** the gold labels and teacher distributions of the public test sets that
 >   `eval/items/`, `eval/reference_rows.json` and `eval/btzsc22/` (BTZSC label texts and targets) include keep their
@@ -199,6 +199,21 @@ learning the teacher's quirks.
 We evaluated Ines-1 on 22 public BTZSC classification datasets (100 examples per dataset), using an extension of the
 jev-benchmarks protocol. Fifteen of the 22 datasets are absent from Ines-1's decision fine-tuning mixture.
 
+| mean macro-F1 (Track A, native interface) | Ines-1 | Julia-1 | laya-multilingual | GLiNER2.5-multi | DiffusionGemma-26B-A4B¹ |
+|---|---:|---:|---:|---:|---:|
+| parameters (total / active per token) | 1.59B / 405M | 144M | 322M | 287M | ~26B / ~4B |
+| **all 22 datasets** | 0.474 | 0.525 | 0.530 | 0.537 | **0.799** |
+| 2 labels (13) | 0.562 | 0.583 | 0.722 | 0.627 | **0.897** |
+| 3–20 labels (4) | 0.489 | 0.634 | 0.559 | 0.541 | **0.695** |
+| 21–72 labels (5) | 0.234 | 0.287 | 0.008 | 0.297 | **0.628** |
+| absent from Ines-1's decision FT mix (15) | 0.486 | 0.536 | 0.515 | 0.528 | **0.765** |
+| difference vs Ines-1, all 22 (95 % CI) | — | +0.051 (−0.026, +0.126) | +0.056 (−0.040, +0.163) | +0.063 (+0.014, +0.107) | +0.326 (+0.253, +0.392) |
+| native ECE, ≤ 20 labels (17 datasets) | 0.127 | 0.298 | 0.093 | 0.159 | 0.116 |
+
+¹ Added on 2026-10-06, after the protocol freeze, on Track A only (FP8 weights served by OpenJev 0.4.0, `think: 0`,
+same request): [`eval/btzsc22/addon_gemma/`](eval/btzsc22/addon_gemma/). The scorer reproduces the other four columns
+exactly. It is a ~16× larger generalist model; speed and memory for the same models are under *Speed* below.
+
 Under the model-specific classification track, mean macro-F1 across datasets was 0.474 for Ines-1, 0.525 for Julia-1,
 0.530 for laya-multilingual, and 0.537 for GLiNER2.5. Using a paired hierarchical bootstrap with datasets as the unit of
 inference, the difference relative to Ines-1 was resolved for GLiNER2.5 (+0.063, 95% CI +0.014 to +0.107), but not for
@@ -244,19 +259,26 @@ Cost: 21,423 domain cases, 23,608 questions per epoch, 28.5M prompt tokens, 2.25
 "Engram removed at inference" = every Engram module returns its input (gate 0) on this checkpoint, which was trained
 with Engram: it measures this model's reliance on Engram, not Engram's contribution to training.
 
-### Speed
+### Speed and memory
+
+[eval/speed.md](eval/speed.md) has the full protocol and all runs. One H200 per model in turn (nothing else on it), HTTP
+closed loop, same requests and load generator for every model (run 3, 2026-10-06):
 
-[eval/speed.md](eval/speed.md) has the full protocol and both runs. One H200, bf16, HTTP closed loop, same requests and
-load generator for every model. Two regimes, kept apart:
+| model | short request¹ c=1 p50 / p95 | short: max questions/s | typed case² c=1 p50 / p95 | typed: max questions/s | GPU memory idle / peak |
+|---|---:|---:|---:|---:|---:|
+| **Ines-1** (1 process, CUDA graphs) | **9.4 / 9.7 ms** | **1,896** | 30.6 / **40.5 ms** | 544 | 9.7 / 21.9 GB |
+| laya-multilingual architecture (4 processes) | 13.0 / 25.9 ms | 1,848 | **20.6** / 89.2 ms | **840** | 8.0 / 24.1 GB |
+| GLiNER2.5-multi architecture (4 workers, no batching) | 48.8 / 49.3 ms | 68 | 123.1 / 129.3 ms | 49 | 6.7 / 26.5 GB |
+| DiffusionGemma-26B-A4B FP8 (OpenJev / vLLM) | 91.7 / 94.0 ms | 79 | 158.8 / 170.2 ms | 143 | 25.8 GiB weights; 120.6 / 135.1 GB reserved³ |
 
-| | single request p50 / p95 | max throughput (questions/s) |
-|---|---:|---:|
-| short request (2 questions, ~60 prompt tokens each) — Ines-1, 1 process | 9.6 / 9.7 ms | ~1,900 |
-| same, 144M / 322M encoders, 4 processes each | 12.7 / 21–26 ms | ~1,700–1,850 |
-| typed-decisions case (5 questions, ~315 tokens each) — Ines-1 | 29.6 / 38.3 ms | ~540 |
-| same, 144M / 322M encoders | 15–16 / 87–88 ms | ~760–1,110 |
+¹ 2 questions, ~60 prompt tokens each (Ines-1 template). ² typed-decisions case, 5 questions, ~300 prompt tokens each
+(Ines-1 template; each server counts tokens differently, see eval/speed.md).
+³ vLLM preallocates KV cache (`gpu_memory_utilization` 0.80); the floor is the weights.
 
-On ~300-token questions **the encoders are 1.4–2.1× faster**.
+Ines-1 has the lowest single-request latency and matches the encoder's peak short-request throughput (within the
+±10 % run-to-run variation); on ~300-token questions the
+322M encoder is faster (1.5× the throughput, lower median) while Ines-1 keeps the tighter tail. The GLiNER row measures
+a server without batching, not the architecture's ceiling. Different serving stacks: not a general speed ranking.
 
 ## Training
 

SHA256SUMS

@@ -8,7 +8,7 @@ cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
 6a7ddcad077839a44697c005ea7e6f455caecaa8054412913108d6e21a1c9dbe  LICENSE-CODE
 121b31536257bd7147d9c9b054f6fb4e2bb68734024415c081ac0b50c7cf48f1  LICENSING_NOTES.md
 1faada60db1ccfc073ae063fc474d9f75cba0c9d2683d03ad66b2074efeb166a  NOTICE
-8b3d5abc94f861cd0c3622a42f3403c7bfb33938009a629b05c88a9fad3f2d2a  README.md
+aed9b754a211decd074a0ddccd654852fc3797a04376bc80b882ea47cc200f58  README.md
 6ec8a2bb55f9331d77b9adbeae7fd71eeb5614a988cd21eee18c7bde775df5ba  TASKSOURCE_LICENSE_AUDIT.md
 1a4df8f10bd30b01e6acc19ef2d935faee5485f0c25452ff64ba6c8598fabd97  THIRD_PARTY_DATA_NOTICE.md
 9708385cf095b543e009b0dd6013aa9985ef68a5b05c5ebef4c14c6a7f97cfa2  assets/architecture.png
@@ -16,6 +16,13 @@ cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
 757003b29ec1d935f236c06ea1e39cf9362c43de358f8b3df74326f8bfdbd40d  config.json
 8f296d2c1dbe85d285bb45cd80d841e733f84168ff130daf825933917684993b  eval/btzsc22/PROTOCOL.md
 5cc7f2cd08668abe6ac5dd5455e61bc2315e556e26c45906150426810fd49455  eval/btzsc22/REPORT.md
+775c02f424c71a5b944ccaf19ba466ad1444644e585cf4785e9fe39d1a0e6140  eval/btzsc22/SHA256SUMS
+cc8ce27ca26efcdc9985323e75fdbad20b602d69b1602c2883a1c80d37dbe15e  eval/btzsc22/addon_gemma/README.md
+80bc6a7b10e75160f88265611a7be4fc07d625980a55697cee3ff3a60dfc5632  eval/btzsc22/addon_gemma/code/run_gemma.py
+d4a2cb890861b2076e79934b5a1dd0f184c5b4418dec4916c8288def58a279e9  eval/btzsc22/addon_gemma/code/score_addon_gemma.py
+7107cfe4a47f306829ee4e30fa50ddeecc9f18f107c5eb300657877b3ce9aded  eval/btzsc22/addon_gemma/per_dataset.csv
+2eca0b816e2ba974b31fd25043ab7ea38b39306f5ed97b16091a6905098a4a2f  eval/btzsc22/addon_gemma/results.json
+5e853fc07f7ab18b695be6b356f8d2c1e6ae23a61e3d79fed0f3be310fe3e26f  eval/btzsc22/addon_gemma/trackA_native.gemma.jsonl
 95a4d4c459940195f03c6b937d750cd8e678527c4e2acaee1f7f2f3a9adc512c  eval/btzsc22/btzsc_test_files.json
 bb7dbba94cc130a09c77226e51cb7b72e5228685fbf8c2f78522290f485eff1e  eval/btzsc22/code/README.md
 afac77fa429f2f37c45e6cf46d0201551e9c98ddf9c01e7ba0cf1ac1feca0ef7  eval/btzsc22/code/btzsc22.yaml
@@ -57,8 +64,10 @@ be0892eb9d9e45f2173e59052d503b26d809253b01bc44a8a70c0e7c48add480  eval/items/v3_
 d0b5cb709b104e8fafcdba8f8c676896ce048c2a009f93c1b19ad3e949fcb128  eval/reference_rows.json
 508a3e33bf77cd4ab9f559d5dcd2ec3e473c30b3b1d54c90a7de5e6490a435a2  eval/report.json
 d5f3f5d214ac20264176a552c7779ddbbee79f97524b64f8c6d1804e1f305ba2  eval/report.md
-6b38584ac67bb856448275789e45bef4bf104fd9462abaec984d232c05dc31ab  eval/speed.md
+2845d32914cc266a9b8a14f20b9ea7b37ad6febf2ae52a084c46ba51af140165  eval/speed.md
 d0b6fcd59367eb58c02eb3b21e7d6c50caa6eccf5d822e8e7ecec22f0e3a275b  eval/speed_raw.log
+da6b8271d68d79a13d4e79e5c5e90e6722d91754bf15e797670a43a6120e99d0  eval/speed_runs/2026-10-06_part1_gemma_laya.txt
+34b9bd9bf06e21f67ad73fc13e8d0deb49efa95a9f181d820b6eced9aa133d6d  eval/speed_runs/2026-10-06_part2_gliner_ines.txt
 22974cc87fd92b5c21168e350ff3db492209a98293ec28a7f5d70396f13c970d  examples/quickstart.py
 6c4fc03dc8ec311323338f6f4f6c0a8a3514f417b57c99f70827d797fe2ead28  mini_v41/__init__.py
 7c57633d0d69fe8a709e0629cbe36b07590d94d0333039c5378fa66df4a74f05  mini_v41/attention.py

eval/btzsc22/SHA256SUMS

@@ -22,3 +22,9 @@ d0ca40988b8ad697223f7ccfc18ac35840c1390a5d46fabc219dab0c02069ed3  ./predictions/
 8f296d2c1dbe85d285bb45cd80d841e733f84168ff130daf825933917684993b  ./PROTOCOL.md
 5cc7f2cd08668abe6ac5dd5455e61bc2315e556e26c45906150426810fd49455  ./REPORT.md
 707f23cd6b4ae30a38573d4f29ce852a82878b23ee062c61a8025d4bae1360ff  ./results.json
+cc8ce27ca26efcdc9985323e75fdbad20b602d69b1602c2883a1c80d37dbe15e  ./addon_gemma/README.md
+80bc6a7b10e75160f88265611a7be4fc07d625980a55697cee3ff3a60dfc5632  ./addon_gemma/code/run_gemma.py
+d4a2cb890861b2076e79934b5a1dd0f184c5b4418dec4916c8288def58a279e9  ./addon_gemma/code/score_addon_gemma.py
+7107cfe4a47f306829ee4e30fa50ddeecc9f18f107c5eb300657877b3ce9aded  ./addon_gemma/per_dataset.csv
+2eca0b816e2ba974b31fd25043ab7ea38b39306f5ed97b16091a6905098a4a2f  ./addon_gemma/results.json
+5e853fc07f7ab18b695be6b356f8d2c1e6ae23a61e3d79fed0f3be310fe3e26f  ./addon_gemma/trackA_native.gemma.jsonl

eval/btzsc22/addon_gemma/README.md

@@ -0,0 +1,39 @@
+# BTZSC-22 add-on: DiffusionGemma-26B-A4B-it (2026-10-06)
+
+**Added after the `btzsc22-ext-v1` freeze.** The protocol, the manifest (2,200 examples) and the scorer are unchanged; this
+directory adds one more model on **Track A (native) only**. Track B (one-vs-rest) was not run for it.
+
+| | |
+|---|---|
+| model | `RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic` (FP8 of Google's DiffusionGemma-26B-A4B-it) |
+| serving | `razorback16/openjev:0.4.0` (vLLM structured reads, `/v1/systemone`), model `openjev-0.1`, `think: 0` (the server default), canvas 64, 1× H200 |
+| request | identical to the other models: `state = {"text": <text>}`, one Choice question `"Which single label best describes the input text?"`, options `label_000…` with the label texts as descriptions |
+| interface limit | 255 options, so all 22 datasets are a **single request** (no knockout) |
+| failures | 0 of 2,200 |
+
+## Files
+
+- `trackA_native.gemma.jsonl`: one prediction per example (full native probability vector). `latency_seconds` was
+  recorded under 16 concurrent clients and is **not** a single-request latency (see `../../speed.md`).
+- `code/run_gemma.py`: the client used.
+- `code/score_addon_gemma.py`: `../code/score_final.py` with Gemma added to the model list. The Track A logic,
+  the paired hierarchical bootstrap and the calibration code are copied verbatim. Rerunning it reproduces the four
+  original models' numbers in `../results.json` exactly (0.474 / 0.525 / 0.530 / 0.537).
+- `results.json`, `per_dataset.csv`: the output. Gemma's Track B fields are `null` / `not_run`.
+
+## Result (Track A, equal-weight mean macro-F1 over datasets)
+
+| group | Ines-1 | Julia-1 | laya-multilingual | GLiNER2.5-multi | DiffusionGemma-26B-A4B |
+|---|---:|---:|---:|---:|---:|
+| all 22 | 0.474 | 0.525 | 0.530 | 0.537 | **0.799** |
+| 2 labels (13) | 0.562 | 0.583 | 0.722 | 0.627 | **0.897** |
+| 3–20 labels (4) | 0.489 | 0.634 | 0.559 | 0.541 | **0.695** |
+| 21–72 labels (5) | 0.234 | 0.287 | 0.008 | 0.297 | **0.628** |
+| absent from Ines-1's decision FT mix (15) | 0.486 | 0.536 | 0.515 | 0.528 | **0.765** |
+
+Gemma − Ines-1, all 22: **+0.326** (95 % CI +0.253 to +0.392, paired hierarchical bootstrap).
+Native calibration (17 datasets with ≤ 20 labels): ECE 0.116 (Ines-1 0.127), Brier 0.250 (Ines-1 0.501),
+coverage at 5 % error budget 0.694 (Ines-1 0.089).
+
+DiffusionGemma-26B-A4B has ~26B total parameters (~4B active per token), about 16× Ines-1's total and 10× its active
+parameters. Speed and memory for the same models are in `../../speed.md` (run 3).

eval/btzsc22/addon_gemma/code/run_gemma.py

@@ -0,0 +1,36 @@
+"""BTZSC-22 backend for DiffusionGemma-26B-A4B-it (FP8) served by OpenJev 0.4.0 (/v1/systemone).
+Same request as run_model.py (state={"text": text}, one Choice, options label_000...), think 0 (OpenJev default),
+native mode only (OpenJev allows 255 options, every dataset fits). argv: url manifest out.jsonl
+"""
+import json, os, sys, time, urllib.request
+from concurrent.futures import ThreadPoolExecutor
+URL, MAN, OUT = sys.argv[1:4]
+Q = "Which single label best describes the input text?"
+
+def call(text, labels):
+    crit = {"label_%03d" % i: l for i, l in enumerate(labels)}
+    body = {"model": "openjev-0.1", "state": {"text": text}, "questions": {"label": {"type": "choice", "instructions": Q, "criteria": crit}}}
+    req = urllib.request.Request(URL, json.dumps(body).encode(), {"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=300) as r:
+        a = json.loads(r.read())["answers"]["label"]
+    return [float(a["probabilities"][k]) for k in crit]
+
+def one(e):
+    t = time.perf_counter()
+    try:
+        p = call(e["text"], e["labels"]); pred = max(range(len(p)), key=p.__getitem__); err = None
+    except Exception as x:  # failures stay in the denominator
+        p, pred, err = None, -1, repr(x)[:300]
+    return {"backend": "gemma", "mode": "native", "group": 0, "dataset": e["dataset"], "example_id": e["example_id"],
+            "target_index": e["target_index"], "predicted_index": pred, "n_labels": len(e["labels"]), "probabilities": p,
+            "calls": 1, "latency_seconds": time.perf_counter() - t, "error": err}
+
+done = set()
+if os.path.exists(OUT):
+    done = {json.loads(l)["example_id"] for l in open(OUT)}
+ex = [e for e in map(json.loads, open(MAN)) if e["example_id"] not in done]
+call(ex[0]["text"], ex[0]["labels"][:2])  # warm-up
+with open(OUT, "a") as f, ThreadPoolExecutor(16) as pool:
+    for r in pool.map(one, ex):
+        f.write(json.dumps(r) + "\n"); f.flush()
+print("done gemma native")

eval/btzsc22/addon_gemma/code/score_addon_gemma.py

@@ -0,0 +1,123 @@
+"""ADD-ON (6-oct, after the freeze): score_final.py + DiffusionGemma-26B-A4B-it (OpenJev) on Track A only. Track A logic, bootstrap and calibration copied verbatim; Track B not run for Gemma.
+Original: btzsc22-ext-v1 analysis (frozen before reading Track B). argv: run_dir(v1) exploratory_dir out_dir
+Run with the gliner venv (uses jev_benchmarks.metrics unchanged)."""
+import csv, json, sys
+from pathlib import Path
+import numpy as np
+from jev_benchmarks.metrics import score_predictions, _macro_f1
+from jev_benchmarks.models import Prediction
+
+V1, EXP, OUT = Path(sys.argv[1]), Path(sys.argv[2]), Path(sys.argv[3]); OUT.mkdir(parents=True, exist_ok=True)
+MODELS = ["ines", "julia", "laya", "gliner", "gemma"]
+GEMMA = Path(__file__).resolve().parent / "runs/btzsc22/addon_gemma/gemma.native.jsonl"
+NAME = {"ines": "Ines-1", "julia": "Julia-1", "laya": "laya-multilingual", "gliner": "GLiNER2.5", "gemma": "DiffusionGemma-26B-A4B-it"}
+IN_MIX = {"amazonpolarity", "banking77", "massive", "wikitoxic_insult", "wikitoxic_obscene", "wikitoxic_threat", "wikitoxic_toxicaggregated"}
+man = [json.loads(l) for l in open(EXP / "manifest.jsonl")]
+DS = list(dict.fromkeys(e["dataset"] for e in man)); K = {e["dataset"]: len(e["labels"]) for e in man}
+IDS = {d: [e["example_id"] for e in man if e["dataset"] == d] for d in DS}
+T = {d: np.array([e["target_index"] for e in man if e["dataset"] == d]) for d in DS}
+LAB = {e["example_id"]: tuple(e["labels"]) for e in man}
+rd = lambda p: {json.loads(l)["example_id"]: json.loads(l) for l in open(p)} if Path(p).exists() else {}
+native = {m: rd(V1 / ("%s.native.jsonl" % m)) for m in MODELS if m != "gemma"}; native["gemma"] = rd(GEMMA)
+ovr = {m: rd(V1 / ("%s.ovr.jsonl" % m)) for m in MODELS}
+ines_k20, ines_k2 = rd(EXP / "ines.k20.jsonl"), rd(EXP / "ines.k2.jsonl")
+
+def preds(src, d):
+    return np.array([(src[i]["predicted_index"] if i in src and not src[i].get("error") else -1) for i in IDS[d]])
+def trackA(m, d):
+    if m == "ines" and K[d] > 26:
+        return preds(ines_k20, d), "knockout-20 (no native > 26 interface)"
+    r = native[m]; s = "single request"
+    if m == "julia" and K[d] > 20: s = "Router(width=20, survivors=2)"
+    return preds(r, d), s
+P = {"A": {m: {d: trackA(m, d)[0] for d in DS} for m in MODELS}, "B": {m: {d: preds(ovr[m], d) for d in DS} for m in MODELS}}
+missing = {tr: {m: int(sum((P[tr][m][d] < 0).sum() for d in DS)) for m in MODELS} for tr in P}
+f1 = lambda tr, m, d, idx=None: _macro_f1(T[d] if idx is None else T[d][idx], P[tr][m][d] if idx is None else P[tr][m][d][idx], K[d])
+acc = lambda tr, m, d: float(np.mean(P[tr][m][d] == T[d]))
+
+GROUPS = {"all 22": DS, "2 labels (13)": [d for d in DS if K[d] == 2], "3-20 labels (4)": [d for d in DS if 2 < K[d] <= 20],
+          "21-72 labels (5)": [d for d in DS if K[d] > 20], "absent from Ines-1 decision FT mixture (15)": [d for d in DS if d not in IN_MIX],
+          "families in Ines-1 decision FT mixture (7)": [d for d in DS if d in IN_MIX]}
+rng = np.random.default_rng(20260917)
+STRATA = {d: [np.where(T[d] == c)[0] for c in np.unique(T[d])] for d in DS}
+def hboot(sel, pairs, B=2000):
+    """paired hierarchical bootstrap: resample datasets, then examples stratified by target; equal-weight mean macro-F1"""
+    out = {p: [] for p in pairs}
+    for _ in range(B):
+        draw = rng.integers(0, len(sel), len(sel))
+        acc_ = {p: [] for p in pairs}
+        for j in draw:
+            d = sel[j]; idx = np.concatenate([rng.choice(c, len(c)) for c in STRATA[d]])
+            cache = {}
+            for p in pairs:
+                (ta, ma), (tb, mb) = p
+                for key in ((ta, ma), (tb, mb)):
+                    if key not in cache: cache[key] = f1(key[0], key[1], d, idx)
+                acc_[p].append(cache[(tb, mb)] - cache[(ta, ma)])
+        for p in pairs: out[p].append(np.mean(acc_[p]))
+    res = {}
+    for p, v in out.items():
+        (ta, ma), (tb, mb) = p
+        obs = float(np.mean([f1(tb, mb, d) - f1(ta, ma, d) for d in sel]))
+        res["%s:%s - %s:%s" % (tb, mb, ta, ma)] = {"difference": obs, "ci95": [float(np.quantile(v, .025)), float(np.quantile(v, .975))]}
+    return res
+
+R = {"protocol": "btzsc22-ext-v1", "missing_predictions": missing, "per_dataset": {}, "global": {}, "bootstrap": {}, "calibration": {},
+     "ines_ablation": {}, "anchors": {}, "ovr_ties": {}}
+for d in DS:
+    R["per_dataset"][d] = {"labels": K[d], "in_ines_ft_mixture": d in IN_MIX,
+                           **{tr: {m: {"accuracy": acc(tr, m, d), "macro_f1": f1(tr, m, d)} for m in MODELS} for tr in ("A", "B")},
+                           "trackA_strategy": {m: trackA(m, d)[1] for m in MODELS}}
+for g, sel in GROUPS.items():
+    R["global"][g] = {tr: {m: float(np.mean([f1(tr, m, d) for d in sel])) for m in MODELS} for tr in ("A", "B")}
+cross = [(("A", "ines"), ("A", m)) for m in MODELS[1:]] + [(("B", "ines"), ("B", m)) for m in MODELS[1:]] + [(("A", m), ("B", m)) for m in MODELS]
+for g, sel in GROUPS.items():
+    R["bootstrap"][g] = hboot(sel, cross, B=2000 if g == "all 22" else 1000)
+
+def calib(srcs, field, sel):
+    out = {}
+    for m in MODELS:
+        per = []
+        for d in sel:
+            rows = []
+            for i in IDS[d]:
+                r = srcs[m].get(i)
+                if not r or r.get("error") or not r.get(field): break
+                rows.append(Prediction(experiment_id="x", backend=m, model_requested=m, model_resolved=m, dataset=d, example_id=i,
+                                       target_index=int(T[d][IDS[d].index(i)]), predicted_index=r["predicted_index"], labels=LAB[i],
+                                       probabilities=tuple(r[field]), latency_seconds=0.0))
+            else:
+                per.append(score_predictions(rows))
+        if per:
+            out[m] = {k: float(np.mean([s[k] for s in per])) for k in ("brier", "nll", "ece", "coverage_at_error_budget", "mean_confidence")} | {"datasets": len(per)}
+    return out
+native_full = [d for d in DS if K[d] <= 20]
+R["calibration"]["native (Track A, full native distributions, <=20 labels, 17 datasets)"] = calib(native, "probabilities", native_full)
+R["calibration"]["derived (Track B softmax of log-odds, same 17 datasets)"] = calib(ovr, "derived_probabilities", native_full)
+R["calibration"]["derived (Track B softmax of log-odds, all 22 datasets)"] = calib(ovr, "derived_probabilities", DS)
+R["ovr_ties"] = {m: {"examples_with_tie_at_max": sum(1 for r in ovr[m].values() if r.get("ties_at_max", 1) > 1), "n": len(ovr[m])} for m in MODELS}
+
+multi = [d for d in DS if K[d] > 2]
+for d in multi:
+    row = {"native(<=26)": acc("A", "ines", d) if K[d] <= 26 else None, "knockout-20": float(np.mean(preds(ines_k20, d) == T[d])) if K[d] > 20 else None,
+           "knockout-2": float(np.mean(preds(ines_k2, d) == T[d])), "one-vs-rest": acc("B", "ines", d)}
+    rowf = {"native(<=26)": _macro_f1(T[d], preds(native["ines"], d), K[d]) if K[d] <= 26 else None,
+            "knockout-20": _macro_f1(T[d], preds(ines_k20, d), K[d]) if K[d] > 20 else None,
+            "knockout-2": _macro_f1(T[d], preds(ines_k2, d), K[d]), "one-vs-rest": f1("B", "ines", d)}
+    R["ines_ablation"][d] = {"labels": K[d], "accuracy": row, "macro_f1": rowf}
+
+pub = {"agnews": 0.70, "emotiondair": 0.44, "banking77": 0.61}
+R["anchors"]["GLiNER2.5 Track A vs published pilot (accuracy)"] = {d: {"ours": acc("A", "gliner", d), "published": v} for d, v in pub.items()}
+card = {"agnews": 0.94, "emotiondair": 0.86, "banking77": 0.64}
+R["anchors"]["Julia-1 Track A vs its model card (accuracy)"] = {d: {"ours": acc("A", "julia", d), "card": v, "ours_strategy": trackA("julia", d)[1]} for d, v in card.items()}
+
+json.dump(R, open(OUT / "results.json", "w"), indent=1)
+with open(OUT / "per_dataset.csv", "w", newline="") as f:
+    w = csv.writer(f); w.writerow(["dataset", "labels", "in_ines1_decision_ft_mixture"] + ["%s_%s_%s" % (tr, m, k) for tr in ("A", "B") for m in MODELS for k in ("accuracy", "macro_f1")] + ["A_strategy_ines", "A_strategy_julia"])
+    for d in DS:
+        x = R["per_dataset"][d]
+        w.writerow([d, K[d], d in IN_MIX] + ["%.4f" % x[tr][m][k] for tr in ("A", "B") for m in MODELS for k in ("accuracy", "macro_f1")] + [x["trackA_strategy"]["ines"], x["trackA_strategy"]["julia"]])
+print("missing", missing)
+for g in GROUPS:
+    print("%-45s A %s | B %s" % (g, "  ".join("%s %.3f" % (m, R["global"][g]["A"][m]) for m in MODELS), "  ".join("%s %.3f" % (m, R["global"][g]["B"][m]) for m in MODELS)))
+for k, v in R["bootstrap"]["all 22"].items(): print("  %-28s %+.3f [%+.3f, %+.3f]" % (k, v["difference"], *v["ci95"]))

eval/btzsc22/addon_gemma/per_dataset.csv

@@ -0,0 +1,23 @@
+dataset,labels,in_ines1_decision_ft_mixture,A_ines_accuracy,A_ines_macro_f1,A_julia_accuracy,A_julia_macro_f1,A_laya_accuracy,A_laya_macro_f1,A_gliner_accuracy,A_gliner_macro_f1,A_gemma_accuracy,A_gemma_macro_f1,B_ines_accuracy,B_ines_macro_f1,B_julia_accuracy,B_julia_macro_f1,B_laya_accuracy,B_laya_macro_f1,B_gliner_accuracy,B_gliner_macro_f1,B_gemma_accuracy,B_gemma_macro_f1,A_strategy_ines,A_strategy_julia
+agnews,4,False,0.5900,0.5843,0.9400,0.9399,0.9400,0.9393,0.7000,0.6589,0.8900,0.8797,0.4200,0.3265,0.4200,0.3995,0.8900,0.8890,0.4500,0.3986,not_run,not_run,single request,single request
+emotiondair,6,False,0.4300,0.3904,0.8600,0.8557,0.3700,0.3527,0.4400,0.4070,0.4000,0.3764,0.3400,0.2948,0.2600,0.2543,0.4300,0.4279,0.3500,0.3193,not_run,not_run,single request,single request
+banking77,72,True,0.3900,0.3427,0.6300,0.5704,0.0200,0.0006,0.6100,0.5693,0.8000,0.7773,0.2900,0.2235,0.0900,0.0472,0.2200,0.1880,0.4100,0.3788,not_run,not_run,knockout-20 (no native > 26 interface),"Router(width=20, survivors=2)"
+amazonpolarity,2,True,0.7800,0.7756,0.6200,0.6162,0.8400,0.8368,0.8800,0.8800,0.9800,0.9800,0.6500,0.6179,0.5800,0.5716,0.9100,0.9098,0.8300,0.8249,not_run,not_run,single request,single request
+imdb,2,False,0.7500,0.7500,0.6800,0.6799,0.7300,0.7149,0.8000,0.7947,0.9700,0.9700,0.6900,0.6808,0.5400,0.5354,0.8100,0.8091,0.7300,0.7175,not_run,not_run,single request,single request
+appreviews,2,False,0.7700,0.7660,0.6500,0.6396,0.7800,0.7756,0.8900,0.8895,0.9400,0.9399,0.8100,0.8095,0.5100,0.4954,0.8900,0.8897,0.8000,0.7960,not_run,not_run,single request,single request
+yelpreviews,2,False,0.7200,0.6997,0.7300,0.7278,0.9000,0.8996,0.9000,0.8994,0.9600,0.9599,0.6800,0.6435,0.5100,0.5088,0.9300,0.9298,0.7400,0.7211,not_run,not_run,single request,single request
+rottentomatoes,2,False,0.5900,0.5671,0.6600,0.6264,0.6900,0.6808,0.7100,0.7064,0.9100,0.9100,0.6400,0.6347,0.4600,0.4325,0.7400,0.7400,0.6700,0.6548,not_run,not_run,single request,single request
+financialphrasebank,3,False,0.5100,0.4891,0.4500,0.3711,0.8100,0.8085,0.6900,0.6821,0.8000,0.8030,0.4500,0.3886,0.2500,0.1744,0.9100,0.9097,0.5500,0.4785,not_run,not_run,single request,single request
+empathetic,32,False,0.2500,0.1909,0.1700,0.1594,0.0600,0.0255,0.3300,0.2807,0.5100,0.4697,0.1800,0.1442,0.0500,0.0390,0.3300,0.2873,0.1400,0.1304,not_run,not_run,knockout-20 (no native > 26 interface),"Router(width=20, survivors=2)"
+biasframes_intent,2,False,0.6000,0.5998,0.5700,0.5665,0.6100,0.5920,0.5800,0.5739,0.8300,0.8300,0.5900,0.5524,0.5900,0.5806,0.6600,0.6550,0.5200,0.3912,not_run,not_run,single request,single request
+massive,59,True,0.3500,0.2502,0.2700,0.1802,0.0100,0.0010,0.4400,0.3827,0.7800,0.7266,0.2000,0.1528,0.0200,0.0122,0.3500,0.3255,0.3000,0.2354,not_run,not_run,knockout-20 (no native > 26 interface),"Router(width=20, survivors=2)"
+yahootopics,10,False,0.5000,0.4915,0.3900,0.3675,0.1500,0.1369,0.4100,0.4170,0.7300,0.7218,0.1800,0.1231,0.1200,0.1127,0.3700,0.3650,0.3200,0.3175,not_run,not_run,single request,single request
+trueteacher,2,False,0.5600,0.5376,0.5200,0.5152,0.5600,0.4658,0.5300,0.5242,0.7100,0.6966,0.5000,0.3333,0.5000,0.4346,0.4800,0.4207,0.5400,0.4524,not_run,not_run,single request,single request
+manifesto,56,False,0.1100,0.0750,0.1000,0.0789,0.0300,0.0077,0.0200,0.0007,0.4400,0.3767,0.0500,0.0329,0.0200,0.0149,0.1100,0.0903,0.0600,0.0136,not_run,not_run,knockout-20 (no native > 26 interface),"Router(width=20, survivors=2)"
+capsotu,21,False,0.3600,0.3121,0.4500,0.4452,0.0500,0.0050,0.3000,0.2524,0.7900,0.7913,0.2100,0.1784,0.1000,0.0799,0.2700,0.2800,0.1700,0.1837,not_run,not_run,single request,"Router(width=20, survivors=2)"
+biasframes_offensive,2,False,0.5200,0.4956,0.5400,0.5119,0.6200,0.6194,0.4600,0.4375,0.8200,0.8199,0.4800,0.4800,0.4500,0.4068,0.6900,0.6885,0.5200,0.3763,not_run,not_run,single request,single request
+biasframes_sex,2,False,0.3400,0.3389,0.5600,0.5556,0.7200,0.7083,0.5300,0.3967,0.9300,0.9298,0.5300,0.5192,0.5600,0.5098,0.7900,0.7900,0.5600,0.4762,not_run,not_run,single request,single request
+wikitoxic_insult,2,True,0.7700,0.7694,0.5600,0.5593,0.8500,0.8482,0.7000,0.6900,0.9300,0.9300,0.6900,0.6900,0.5400,0.5308,0.8000,0.7960,0.5800,0.5000,not_run,not_run,single request,single request
+wikitoxic_obscene,2,True,0.4400,0.3455,0.5600,0.5572,0.6500,0.6419,0.5200,0.3912,0.8800,0.8798,0.7300,0.7300,0.6300,0.6190,0.8300,0.8279,0.6200,0.5703,not_run,not_run,single request,single request
+wikitoxic_threat,2,True,0.4700,0.3498,0.4700,0.4674,0.9100,0.9100,0.5300,0.3967,0.9700,0.9700,0.7300,0.7254,0.5500,0.5423,0.7700,0.7614,0.5200,0.3763,not_run,not_run,single request,single request
+wikitoxic_toxicaggregated,2,True,0.3300,0.3049,0.5700,0.5502,0.7000,0.6921,0.6200,0.5766,0.8500,0.8500,0.7700,0.7681,0.5700,0.5572,0.8700,0.8678,0.6200,0.5766,not_run,not_run,single request,single request

eval/btzsc22/addon_gemma/results.json

@@ -0,0 +1,2044 @@
+{
+ "protocol": "btzsc22-ext-v1",
+ "missing_predictions": {
+  "A": {
+   "ines": 0,
+   "julia": 0,
+   "laya": 0,
+   "gliner": 0,
+   "gemma": 0
+  },
+  "B": {
+   "ines": 0,
+   "julia": 0,
+   "laya": 0,
+   "gliner": 0,
+   "gemma": "not run (Track B was not run for Gemma)"
+  }
+ },
+ "per_dataset": {
+  "agnews": {
+   "labels": 4,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.59,
+     "macro_f1": 0.5843451916892106
+    },
+    "julia": {
+     "accuracy": 0.94,
+     "macro_f1": 0.9399173646545361
+    },
+    "laya": {
+     "accuracy": 0.94,
+     "macro_f1": 0.939271525304066
+    },
+    "gliner": {
+     "accuracy": 0.7,
+     "macro_f1": 0.6588522320524076
+    },
+    "gemma": {
+     "accuracy": 0.89,
+     "macro_f1": 0.8797325209206743
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.42,
+     "macro_f1": 0.32652938961350175
+    },
+    "julia": {
+     "accuracy": 0.42,
+     "macro_f1": 0.399474348626891
+    },
+    "laya": {
+     "accuracy": 0.89,
+     "macro_f1": 0.8889659795261577
+    },
+    "gliner": {
+     "accuracy": 0.45,
+     "macro_f1": 0.39861358684888093
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "emotiondair": {
+   "labels": 6,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.43,
+     "macro_f1": 0.39042230395613853
+    },
+    "julia": {
+     "accuracy": 0.86,
+     "macro_f1": 0.8556934538891329
+    },
+    "laya": {
+     "accuracy": 0.37,
+     "macro_f1": 0.35268000920174836
+    },
+    "gliner": {
+     "accuracy": 0.44,
+     "macro_f1": 0.4070167592505414
+    },
+    "gemma": {
+     "accuracy": 0.4,
+     "macro_f1": 0.37637917637917634
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.34,
+     "macro_f1": 0.29478472669962036
+    },
+    "julia": {
+     "accuracy": 0.26,
+     "macro_f1": 0.25425384800384804
+    },
+    "laya": {
+     "accuracy": 0.43,
+     "macro_f1": 0.4279340742423987
+    },
+    "gliner": {
+     "accuracy": 0.35,
+     "macro_f1": 0.3193197964297709
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "banking77": {
+   "labels": 72,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.39,
+     "macro_f1": 0.34265873015873016
+    },
+    "julia": {
+     "accuracy": 0.63,
+     "macro_f1": 0.5703703703703703
+    },
+    "laya": {
+     "accuracy": 0.02,
+     "macro_f1": 0.0005611672278338945
+    },
+    "gliner": {
+     "accuracy": 0.61,
+     "macro_f1": 0.569268077601411
+    },
+    "gemma": {
+     "accuracy": 0.8,
+     "macro_f1": 0.7773148148148148
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.29,
+     "macro_f1": 0.2234898589065256
+    },
+    "julia": {
+     "accuracy": 0.09,
+     "macro_f1": 0.04716343327454439
+    },
+    "laya": {
+     "accuracy": 0.22,
+     "macro_f1": 0.18796296296296294
+    },
+    "gliner": {
+     "accuracy": 0.41,
+     "macro_f1": 0.3787818662818662
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "knockout-20 (no native > 26 interface)",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "amazonpolarity": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.78,
+     "macro_f1": 0.7756017951856384
+    },
+    "julia": {
+     "accuracy": 0.62,
+     "macro_f1": 0.6161616161616161
+    },
+    "laya": {
+     "accuracy": 0.84,
+     "macro_f1": 0.8368013055895553
+    },
+    "gliner": {
+     "accuracy": 0.88,
+     "macro_f1": 0.8799519807923168
+    },
+    "gemma": {
+     "accuracy": 0.98,
+     "macro_f1": 0.98
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.65,
+     "macro_f1": 0.6178622120318812
+    },
+    "julia": {
+     "accuracy": 0.58,
+     "macro_f1": 0.5716034271725826
+    },
+    "laya": {
+     "accuracy": 0.91,
+     "macro_f1": 0.9097744360902256
+    },
+    "gliner": {
+     "accuracy": 0.83,
+     "macro_f1": 0.8249407887962105
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "imdb": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.75,
+     "macro_f1": 0.74997499749975
+    },
+    "julia": {
+     "accuracy": 0.68,
+     "macro_f1": 0.6798719487795117
+    },
+    "laya": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7149192271143491
+    },
+    "gliner": {
+     "accuracy": 0.8,
+     "macro_f1": 0.7947454844006567
+    },
+    "gemma": {
+     "accuracy": 0.97,
+     "macro_f1": 0.96999699969997
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6807743795695603
+    },
+    "julia": {
+     "accuracy": 0.54,
+     "macro_f1": 0.5353535353535354
+    },
+    "laya": {
+     "accuracy": 0.81,
+     "macro_f1": 0.8090644156366193
+    },
+    "gliner": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7175436761167486
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "appreviews": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.77,
+     "macro_f1": 0.7660461804495982
+    },
+    "julia": {
+     "accuracy": 0.65,
+     "macro_f1": 0.6395839769333744
+    },
+    "laya": {
+     "accuracy": 0.78,
+     "macro_f1": 0.7756017951856384
+    },
+    "gliner": {
+     "accuracy": 0.89,
+     "macro_f1": 0.8894583458948849
+    },
+    "gemma": {
+     "accuracy": 0.94,
+     "macro_f1": 0.9399038461538461
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.81,
+     "macro_f1": 0.8095238095238095
+    },
+    "julia": {
+     "accuracy": 0.51,
+     "macro_f1": 0.49541756770672435
+    },
+    "laya": {
+     "accuracy": 0.89,
+     "macro_f1": 0.8897243107769424
+    },
+    "gliner": {
+     "accuracy": 0.8,
+     "macro_f1": 0.7960016319869441
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "yelpreviews": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.72,
+     "macro_f1": 0.6996996996996997
+    },
+    "julia": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7277951406391774
+    },
+    "laya": {
+     "accuracy": 0.9,
+     "macro_f1": 0.8996386993175431
+    },
+    "gliner": {
+     "accuracy": 0.9,
+     "macro_f1": 0.8993558776167472
+    },
+    "gemma": {
+     "accuracy": 0.96,
+     "macro_f1": 0.9599358974358975
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.68,
+     "macro_f1": 0.64349376114082
+    },
+    "julia": {
+     "accuracy": 0.51,
+     "macro_f1": 0.5087719298245614
+    },
+    "laya": {
+     "accuracy": 0.93,
+     "macro_f1": 0.9298245614035088
+    },
+    "gliner": {
+     "accuracy": 0.74,
+     "macro_f1": 0.7211497211497211
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "rottentomatoes": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.59,
+     "macro_f1": 0.5670995670995671
+    },
+    "julia": {
+     "accuracy": 0.66,
+     "macro_f1": 0.6263736263736264
+    },
+    "laya": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6807743795695603
+    },
+    "gliner": {
+     "accuracy": 0.71,
+     "macro_f1": 0.7064480210547626
+    },
+    "gemma": {
+     "accuracy": 0.91,
+     "macro_f1": 0.90999099909991
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.64,
+     "macro_f1": 0.6347402597402598
+    },
+    "julia": {
+     "accuracy": 0.46,
+     "macro_f1": 0.43253467843631777
+    },
+    "laya": {
+     "accuracy": 0.74,
+     "macro_f1": 0.74
+    },
+    "gliner": {
+     "accuracy": 0.67,
+     "macro_f1": 0.6547756041426928
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "financialphrasebank": {
+   "labels": 3,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.51,
+     "macro_f1": 0.48911853494169083
+    },
+    "julia": {
+     "accuracy": 0.45,
+     "macro_f1": 0.37110209601081806
+    },
+    "laya": {
+     "accuracy": 0.81,
+     "macro_f1": 0.8084906388321863
+    },
+    "gliner": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6821411017203141
+    },
+    "gemma": {
+     "accuracy": 0.8,
+     "macro_f1": 0.8029584462511292
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.45,
+     "macro_f1": 0.388558378810103
+    },
+    "julia": {
+     "accuracy": 0.25,
+     "macro_f1": 0.17441737846672992
+    },
+    "laya": {
+     "accuracy": 0.91,
+     "macro_f1": 0.9096611274030629
+    },
+    "gliner": {
+     "accuracy": 0.55,
+     "macro_f1": 0.47845237675746155
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "empathetic": {
+   "labels": 32,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.25,
+     "macro_f1": 0.1909003551273288
+    },
+    "julia": {
+     "accuracy": 0.17,
+     "macro_f1": 0.15942460317460316
+    },
+    "laya": {
+     "accuracy": 0.06,
+     "macro_f1": 0.02546768707482993
+    },
+    "gliner": {
+     "accuracy": 0.33,
+     "macro_f1": 0.2807043650793651
+    },
+    "gemma": {
+     "accuracy": 0.51,
+     "macro_f1": 0.46972402597402596
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.18,
+     "macro_f1": 0.14416745535166586
+    },
+    "julia": {
+     "accuracy": 0.05,
+     "macro_f1": 0.039042207792207795
+    },
+    "laya": {
+     "accuracy": 0.33,
+     "macro_f1": 0.2872519841269841
+    },
+    "gliner": {
+     "accuracy": 0.14,
+     "macro_f1": 0.13043154761904763
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "knockout-20 (no native > 26 interface)",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "biasframes_intent": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.6,
+     "macro_f1": 0.5998399359743898
+    },
+    "julia": {
+     "accuracy": 0.57,
+     "macro_f1": 0.5664885573142454
+    },
+    "laya": {
+     "accuracy": 0.61,
+     "macro_f1": 0.5920075321686369
+    },
+    "gliner": {
+     "accuracy": 0.58,
+     "macro_f1": 0.5738636363636364
+    },
+    "gemma": {
+     "accuracy": 0.83,
+     "macro_f1": 0.8299829982998299
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.59,
+     "macro_f1": 0.5523528769516323
+    },
+    "julia": {
+     "accuracy": 0.59,
+     "macro_f1": 0.5805626598465473
+    },
+    "laya": {
+     "accuracy": 0.66,
+     "macro_f1": 0.6550324675324675
+    },
+    "gliner": {
+     "accuracy": 0.52,
+     "macro_f1": 0.39117199391172
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "massive": {
+   "labels": 59,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.35,
+     "macro_f1": 0.2502433536331841
+    },
+    "julia": {
+     "accuracy": 0.27,
+     "macro_f1": 0.1801721818670971
+    },
+    "laya": {
+     "accuracy": 0.01,
+     "macro_f1": 0.0009685230024213075
+    },
+    "gliner": {
+     "accuracy": 0.44,
+     "macro_f1": 0.38272522334474407
+    },
+    "gemma": {
+     "accuracy": 0.78,
+     "macro_f1": 0.7265536723163842
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.2,
+     "macro_f1": 0.15282852740479855
+    },
+    "julia": {
+     "accuracy": 0.02,
+     "macro_f1": 0.01224105461393597
+    },
+    "laya": {
+     "accuracy": 0.35,
+     "macro_f1": 0.3254640839386602
+    },
+    "gliner": {
+     "accuracy": 0.3,
+     "macro_f1": 0.2353510895883777
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "knockout-20 (no native > 26 interface)",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "yahootopics": {
+   "labels": 10,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.5,
+     "macro_f1": 0.4915120854091442
+    },
+    "julia": {
+     "accuracy": 0.39,
+     "macro_f1": 0.3675051008447593
+    },
+    "laya": {
+     "accuracy": 0.15,
+     "macro_f1": 0.13690074545337702
+    },
+    "gliner": {
+     "accuracy": 0.41,
+     "macro_f1": 0.4169864449276214
+    },
+    "gemma": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7217792466130062
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.18,
+     "macro_f1": 0.12307189542483658
+    },
+    "julia": {
+     "accuracy": 0.12,
+     "macro_f1": 0.11272908345326083
+    },
+    "laya": {
+     "accuracy": 0.37,
+     "macro_f1": 0.36499120762278653
+    },
+    "gliner": {
+     "accuracy": 0.32,
+     "macro_f1": 0.31750090466808734
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "trueteacher": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.56,
+     "macro_f1": 0.537620849096259
+    },
+    "julia": {
+     "accuracy": 0.52,
+     "macro_f1": 0.5151515151515151
+    },
+    "laya": {
+     "accuracy": 0.56,
+     "macro_f1": 0.46576007770762506
+    },
+    "gliner": {
+     "accuracy": 0.53,
+     "macro_f1": 0.5242433444680635
+    },
+    "gemma": {
+     "accuracy": 0.71,
+     "macro_f1": 0.69662098545873
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.5,
+     "macro_f1": 0.3333333333333333
+    },
+    "julia": {
+     "accuracy": 0.5,
+     "macro_f1": 0.43464495703301675
+    },
+    "laya": {
+     "accuracy": 0.48,
+     "macro_f1": 0.4206773618538324
+    },
+    "gliner": {
+     "accuracy": 0.54,
+     "macro_f1": 0.45238095238095233
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "manifesto": {
+   "labels": 56,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.11,
+     "macro_f1": 0.07504578754578754
+    },
+    "julia": {
+     "accuracy": 0.1,
+     "macro_f1": 0.0788690476190476
+    },
+    "laya": {
+     "accuracy": 0.03,
+     "macro_f1": 0.007711038961038961
+    },
+    "gliner": {
+     "accuracy": 0.02,
+     "macro_f1": 0.0007072135785007072
+    },
+    "gemma": {
+     "accuracy": 0.44,
+     "macro_f1": 0.37665816326530605
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.05,
+     "macro_f1": 0.032902735562310034
+    },
+    "julia": {
+     "accuracy": 0.02,
+     "macro_f1": 0.01488095238095238
+    },
+    "laya": {
+     "accuracy": 0.11,
+     "macro_f1": 0.09031507339778015
+    },
+    "gliner": {
+     "accuracy": 0.06,
+     "macro_f1": 0.013575543120473996
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "knockout-20 (no native > 26 interface)",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "capsotu": {
+   "labels": 21,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.36,
+     "macro_f1": 0.3120540695728665
+    },
+    "julia": {
+     "accuracy": 0.45,
+     "macro_f1": 0.4451588094445238
+    },
+    "laya": {
+     "accuracy": 0.05,
+     "macro_f1": 0.005012531328320802
+    },
+    "gliner": {
+     "accuracy": 0.3,
+     "macro_f1": 0.2524050024050024
+    },
+    "gemma": {
+     "accuracy": 0.79,
+     "macro_f1": 0.7912510769653627
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.21,
+     "macro_f1": 0.17844985702128557
+    },
+    "julia": {
+     "accuracy": 0.1,
+     "macro_f1": 0.0798883656026513
+    },
+    "laya": {
+     "accuracy": 0.27,
+     "macro_f1": 0.2799847801788137
+    },
+    "gliner": {
+     "accuracy": 0.17,
+     "macro_f1": 0.18369192344347623
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "biasframes_offensive": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.52,
+     "macro_f1": 0.49558638083228246
+    },
+    "julia": {
+     "accuracy": 0.54,
+     "macro_f1": 0.5118845500848896
+    },
+    "laya": {
+     "accuracy": 0.62,
+     "macro_f1": 0.6193910256410255
+    },
+    "gliner": {
+     "accuracy": 0.46,
+     "macro_f1": 0.4375
+    },
+    "gemma": {
+     "accuracy": 0.82,
+     "macro_f1": 0.8199279711884754
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.48,
+     "macro_f1": 0.48
+    },
+    "julia": {
+     "accuracy": 0.45,
+     "macro_f1": 0.40675223816201056
+    },
+    "laya": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6884735202492211
+    },
+    "gliner": {
+     "accuracy": 0.52,
+     "macro_f1": 0.37629937629937626
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "biasframes_sex": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.34,
+     "macro_f1": 0.3389423076923077
+    },
+    "julia": {
+     "accuracy": 0.56,
+     "macro_f1": 0.5555555555555556
+    },
+    "laya": {
+     "accuracy": 0.72,
+     "macro_f1": 0.7083333333333334
+    },
+    "gliner": {
+     "accuracy": 0.53,
+     "macro_f1": 0.39673982800667434
+    },
+    "gemma": {
+     "accuracy": 0.93,
+     "macro_f1": 0.9298245614035088
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.53,
+     "macro_f1": 0.5191815856777494
+    },
+    "julia": {
+     "accuracy": 0.56,
+     "macro_f1": 0.5098039215686274
+    },
+    "laya": {
+     "accuracy": 0.79,
+     "macro_f1": 0.78997899789979
+    },
+    "gliner": {
+     "accuracy": 0.56,
+     "macro_f1": 0.47619047619047616
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "wikitoxic_insult": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.77,
+     "macro_f1": 0.7694235588972431
+    },
+    "julia": {
+     "accuracy": 0.56,
+     "macro_f1": 0.5592948717948718
+    },
+    "laya": {
+     "accuracy": 0.85,
+     "macro_f1": 0.8481627695110842
+    },
+    "gliner": {
+     "accuracy": 0.7,
+     "macro_f1": 0.6899545266639107
+    },
+    "gemma": {
+     "accuracy": 0.93,
+     "macro_f1": 0.9299929992999301
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6899689968996899
+    },
+    "julia": {
+     "accuracy": 0.54,
+     "macro_f1": 0.5308037535699714
+    },
+    "laya": {
+     "accuracy": 0.8,
+     "macro_f1": 0.7960016319869441
+    },
+    "gliner": {
+     "accuracy": 0.58,
+     "macro_f1": 0.5
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "wikitoxic_obscene": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.44,
+     "macro_f1": 0.34548854604955587
+    },
+    "julia": {
+     "accuracy": 0.56,
+     "macro_f1": 0.5571658615136876
+    },
+    "laya": {
+     "accuracy": 0.65,
+     "macro_f1": 0.6419437340153453
+    },
+    "gliner": {
+     "accuracy": 0.52,
+     "macro_f1": 0.39117199391172
+    },
+    "gemma": {
+     "accuracy": 0.88,
+     "macro_f1": 0.8798076923076923
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.73,
+     "macro_f1": 0.72997299729973
+    },
+    "julia": {
+     "accuracy": 0.63,
+     "macro_f1": 0.6189887756152817
+    },
+    "laya": {
+     "accuracy": 0.83,
+     "macro_f1": 0.8279178054458953
+    },
+    "gliner": {
+     "accuracy": 0.62,
+     "macro_f1": 0.5703301673450927
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "wikitoxic_threat": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.47,
+     "macro_f1": 0.3497730339835603
+    },
+    "julia": {
+     "accuracy": 0.47,
+     "macro_f1": 0.467390212038991
+    },
+    "laya": {
+     "accuracy": 0.91,
+     "macro_f1": 0.90999099909991
+    },
+    "gliner": {
+     "accuracy": 0.53,
+     "macro_f1": 0.39673982800667434
+    },
+    "gemma": {
+     "accuracy": 0.97,
+     "macro_f1": 0.96999699969997
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7253585596582239
+    },
+    "julia": {
+     "accuracy": 0.55,
+     "macro_f1": 0.54226426609704
+    },
+    "laya": {
+     "accuracy": 0.77,
+     "macro_f1": 0.7613860358958398
+    },
+    "gliner": {
+     "accuracy": 0.52,
+     "macro_f1": 0.37629937629937626
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  },
+  "wikitoxic_toxicaggregated": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.33,
+     "macro_f1": 0.30490714804440294
+    },
+    "julia": {
+     "accuracy": 0.57,
+     "macro_f1": 0.5501621508525996
+    },
+    "laya": {
+     "accuracy": 0.7,
+     "macro_f1": 0.6921182266009853
+    },
+    "gliner": {
+     "accuracy": 0.62,
+     "macro_f1": 0.5766488413547237
+    },
+    "gemma": {
+     "accuracy": 0.85,
+     "macro_f1": 0.84998499849985
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.77,
+     "macro_f1": 0.7681217864704104
+    },
+    "julia": {
+     "accuracy": 0.57,
+     "macro_f1": 0.557203171661003
+    },
+    "laya": {
+     "accuracy": 0.87,
+     "macro_f1": 0.8677652324280338
+    },
+    "gliner": {
+     "accuracy": 0.62,
+     "macro_f1": 0.5766488413547237
+    },
+    "gemma": null
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request",
+    "gemma": "single request"
+   }
+  }
+ },
+ "global": {
+  "all 22": {
+   "A": {
+    "ines": 0.4739229278426516,
+    "julia": 0.5245951186849342,
+    "laya": 0.5301139532382007,
+    "gliner": 0.536710369477031,
+    "gemma": 0.7994690041839767
+   },
+   "B": {
+    "ines": 0.45679397195871585,
+    "julia": 0.3572179797391928,
+    "laya": 0.6294614568454058,
+    "gliner": 0.44952051094233986,
+    "gemma": null
+   }
+  },
+  "2 labels (13)": {
+   "A": {
+    "ines": 0.561538769269558,
+    "julia": 0.5825291987072047,
+    "laya": 0.7219571619118916,
+    "gliner": 0.627447823733444,
+    "gemma": 0.8973820729652007
+   },
+   "B": {
+    "ines": 0.6295911198690077,
+    "julia": 0.5172849909267092,
+    "laya": 0.7758169828614861,
+    "gliner": 0.5718255850749258,
+    "gemma": null
+   }
+  },
+  "3-20 labels (4)": {
+   "A": {
+    "ines": 0.48884952899904605,
+    "julia": 0.6335545038498116,
+    "laya": 0.5593357296978444,
+    "gliner": 0.5412491344877212,
+    "gemma": 0.6952123475409966
+   },
+   "B": {
+    "ines": 0.28323609763701546,
+    "julia": 0.23521866463768246,
+    "laya": 0.6478880971986014,
+    "gliner": 0.3784716661760502,
+    "gemma": null
+   }
+  },
+  "21-72 labels (5)": {
+   "A": {
+    "ines": 0.23418045920757943,
+    "julia": 0.28679900249512835,
+    "laya": 0.00794418951888898,
+    "gliner": 0.29716197640180464,
+    "gemma": 0.6283003506671788
+   },
+   "B": {
+    "ines": 0.14636768684931714,
+    "julia": 0.03864320273285837,
+    "laya": 0.23419577692104018,
+    "gliner": 0.18836639401064836,
+    "gemma": null
+   }
+  },
+  "absent from Ines-1 decision FT mixture (15)": {
+   "A": {
+    "ines": 0.4858805497724014,
+    "julia": 0.5360250230979545,
+    "laya": 0.5154640164128853,
+    "gliner": 0.5280778437879452,
+    "gemma": 0.7649777943405899
+   },
+   "B": {
+    "ines": 0.4094576296280325,
+    "julia": 0.3319018448171922,
+    "laya": 0.6114586574566909,
+    "gliner": 0.4284732740710553,
+    "gemma": null
+   }
+  },
+  "families in Ines-1 decision FT mixture (7)": {
+   "A": {
+    "ines": 0.44829945227890217,
+    "julia": 0.500102466371319,
+    "laya": 0.5615066750067336,
+    "gliner": 0.5552086388107859,
+    "gemma": 0.8733787395626631
+   },
+   "B": {
+    "ines": 0.5582289912387514,
+    "julia": 0.411466840286337,
+    "laya": 0.6680388841069373,
+    "gliner": 0.4946217328093781,
+    "gemma": null
+   }
+  }
+ },
+ "bootstrap": {
+  "all 22": {
+   "A:julia - A:ines": {
+    "difference": 0.050672190842282465,
+    "ci95": [
+     -0.026197789339992814,
+     0.12574349835362167
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.05619102539554903,
+    "ci95": [
+     -0.03957741468351192,
+     0.16332715413710902
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.06278744163437923,
+    "ci95": [
+     0.014208626140706566,
+     0.10706083324209556
+    ]
+   },
+   "A:gemma - A:ines": {
+    "difference": 0.32554607634132515,
+    "ci95": [
+     0.2531753321872848,
+     0.39198685847503467
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.099575992219523,
+    "ci95": [
+     -0.14637550498902815,
+     -0.05017137416536893
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.17266748488669,
+    "ci95": [
+     0.11284488099527132,
+     0.23744000375671462
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": -0.007273461016375937,
+    "ci95": [
+     -0.06982555475121606,
+     0.05100715361862577
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": -0.017128955883935842,
+    "ci95": [
+     -0.09913170297625284,
+     0.07045906519324419
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.16737713894574127,
+    "ci95": [
+     -0.2550239493204071,
+     -0.08802480974576297
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.09934750360720516,
+    "ci95": [
+     0.0454206858231911,
+     0.14433272375703962
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.087189858534691,
+    "ci95": [
+     -0.13204415683257073,
+     -0.0406803117967699
+    ]
+   }
+  },
+  "2 labels (13)": {
+   "A:julia - A:ines": {
+    "difference": 0.020990429437646712,
+    "ci95": [
+     -0.06061145608979183,
+     0.10740818043124331
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.16041839264233365,
+    "ci95": [
+     0.059458802921318936,
+     0.27206775011759543
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.0659090544638859,
+    "ci95": [
+     0.004675393764386895,
+     0.12582906809147268
+    ]
+   },
+   "A:gemma - A:ines": {
+    "difference": 0.33584330369564275,
+    "ci95": [
+     0.25214648887513785,
+     0.4349238101458538
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.11230612894229851,
+    "ci95": [
+     -0.18080307502401038,
+     -0.042581067377537536
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.14622586299247847,
+    "ci95": [
+     0.09520841847771228,
+     0.20567775456268422
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": -0.057765534794081974,
+    "ci95": [
+     -0.1428996478267053,
+     0.029876283921421498
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": 0.06805235059944967,
+    "ci95": [
+     -0.04906521051519053,
+     0.19755199873107607
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.06524420778049556,
+    "ci95": [
+     -0.12548066475124145,
+     -0.012510733064128867
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.05385982094959447,
+    "ci95": [
+     -0.0009695761275055773,
+     0.1065780484194153
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.0556222386585182,
+    "ci95": [
+     -0.11618959302529945,
+     0.007571405517821818
+    ]
+   }
+  },
+  "3-20 labels (4)": {
+   "A:julia - A:ines": {
+    "difference": 0.14470497485076556,
+    "ci95": [
+     -0.12895039644581718,
+     0.42568245069405136
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.07048620069879838,
+    "ci95": [
+     -0.229964039114142,
+     0.3531563180395061
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.052399605488675075,
+    "ci95": [
+     -0.05581876515491288,
+     0.1600753585487263
+    ]
+   },
+   "A:gemma - A:ines": {
+    "difference": 0.2063628185419505,
+    "ci95": [
+     0.06159339629875036,
+     0.3329951554044995
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.048017432999332976,
+    "ci95": [
+     -0.16164512046595278,
+     0.05754769356829411
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.364651999561586,
+    "ci95": [
+     0.18329205393405798,
+     0.5540391760322673
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": 0.09523556853903474,
+    "ci95": [
+     0.01900107622902177,
+     0.17430955225269407
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": -0.2056134313620306,
+    "ci95": [
+     -0.3164880910714138,
+     -0.09244502441135767
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.3983358392121291,
+    "ci95": [
+     -0.5756158997267588,
+     -0.20827382157801752
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.08855236750075707,
+    "ci95": [
+     -0.008752159824462086,
+     0.1939655463264431
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.16277746831167095,
+    "ci95": [
+     -0.25637377988986315,
+     -0.07162803351218687
+    ]
+   }
+  },
+  "21-72 labels (5)": {
+   "A:julia - A:ines": {
+    "difference": 0.05261854328754896,
+    "ci95": [
+     -0.04301551177445001,
+     0.15765204279382006
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": -0.22623626968869046,
+    "ci95": [
+     -0.30032025667320744,
+     -0.12187817221877785
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.06298151719422522,
+    "ci95": [
+     -0.039063579916225874,
+     0.1689149673736492
+    ]
+   },
+   "A:gemma - A:ines": {
+    "difference": 0.39411989145959925,
+    "ci95": [
+     0.3019916275555586,
+     0.473337812412976
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.10772448411645877,
+    "ci95": [
+     -0.15259487999276594,
+     -0.04920310152464395
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.0878280900717231,
+    "ci95": [
+     0.011100563313016499,
+     0.14839732102914302
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": 0.041998707161331236,
+    "ci95": [
+     -0.019529909671919345,
+     0.11057761360469363
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": -0.08781277235826232,
+    "ci95": [
+     -0.12624266008952723,
+     -0.04849955912024958
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.24815579976227004,
+    "ci95": [
+     -0.3988206525705068,
+     -0.09861247338677506
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.22625158740215126,
+    "ci95": [
+     0.12816497504348637,
+     0.286499258070288
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.1087955823911563,
+    "ci95": [
+     -0.1700071752364474,
+     -0.032048604061386245
+    ]
+   }
+  },
+  "absent from Ines-1 decision FT mixture (15)": {
+   "A:julia - A:ines": {
+    "difference": 0.050144473325553045,
+    "ci95": [
+     -0.028885054771255515,
+     0.1500566532935792
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.029583466640483884,
+    "ci95": [
+     -0.08075161791194013,
+     0.14421513826606108
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.04219729401554383,
+    "ci95": [
+     -0.006600902674686908,
+     0.09437438464034366
+    ]
+   },
+   "A:gemma - A:ines": {
+    "difference": 0.2790972445681884,
+    "ci95": [
+     0.211381616754686,
+     0.3521485454260135
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.0775557848108404,
+    "ci95": [
+     -0.14064220126958463,
+     -0.013517268561773333
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.2020010278286585,
+    "ci95": [
+     0.1269579464717378,
+     0.2900629696566373
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": 0.019015644443022794,
+    "ci95": [
+     -0.03174972794352044,
+     0.07223435697834166
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": -0.0764229201443689,
+    "ci95": [
+     -0.15017289217507543,
+     -0.009184764537925973
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.20412317828076224,
+    "ci95": [
+     -0.2998785466263466,
+     -0.11592881332069362
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.09599464104380576,
+    "ci95": [
+     0.0459047217540702,
+     0.1442013728576654
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.09960456971688988,
+    "ci95": [
+     -0.14649500828750583,
+     -0.04960181354566028
+    ]
+   }
+  },
+  "families in Ines-1 decision FT mixture (7)": {
+   "A:julia - A:ines": {
+    "difference": 0.051803014092416944,
+    "ci95": [
+     -0.10019425716510703,
+     0.1778425425557031
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.11320722272783149,
+    "ci95": [
+     -0.12479087223953271,
+     0.3295312555301822
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.10690918653188367,
+    "ci95": [
+     0.019280206335977032,
+     0.19106643449909544
+    ]
+   },
+   "A:gemma - A:ines": {
+    "difference": 0.42507928728376093,
+    "ci95": [
+     0.28040693225491636,
+     0.5352594526522105
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.14676215095241435,
+    "ci95": [
+     -0.1986636251903599,
+     -0.08740265140015356
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.10980989286818602,
+    "ci95": [
+     0.03243625252632493,
+     0.19414880936010162
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": -0.06360725842937322,
+    "ci95": [
+     -0.21886027228587646,
+     0.08094216222892091
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": 0.10992953895984924,
+    "ci95": [
+     -0.07562649281899102,
+     0.2927199100383737
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.08863562608498207,
+    "ci95": [
+     -0.24495270619294315,
+     0.033542744774361914
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.10653220910020378,
+    "ci95": [
+     -0.015590406007179902,
+     0.20797413933948058
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.06058690600140765,
+    "ci95": [
+     -0.14995189706107254,
+     0.032545729267887305
+    ]
+   }
+  }
+ },
+ "calibration": {
+  "native (Track A, full native distributions, <=20 labels, 17 datasets)": {
+   "ines": {
+    "brier": 0.5014338202978408,
+    "nll": 0.8000206904458703,
+    "ece": 0.1270706365126021,
+    "coverage_at_error_budget": 0.08941176470588236,
+    "mean_confidence": 0.6118652016888647,
+    "datasets": 17
+   },
+   "julia": {
+    "brier": 0.645280763707045,
+    "nll": 1.6618435294062641,
+    "ece": 0.2983721069531266,
+    "coverage_at_error_budget": 0.1288235294117647,
+    "mean_confidence": 0.8941619627630315,
+    "datasets": 17
+   },
+   "laya": {
+    "brier": 0.4006447652764705,
+    "nll": 0.6925248427789752,
+    "ece": 0.09332229411764704,
+    "coverage_at_error_budget": 0.25588235294117645,
+    "mean_confidence": 0.6874152352941176,
+    "datasets": 17
+   },
+   "gliner": {
+    "brier": 0.48542838104085484,
+    "nll": 0.8599173300405782,
+    "ece": 0.1585389382487745,
+    "coverage_at_error_budget": 0.19705882352941173,
+    "mean_confidence": 0.7310719386879542,
+    "datasets": 17
+   },
+   "gemma": {
+    "brier": 0.24999771502464618,
+    "nll": 0.6128898920857043,
+    "ece": 0.1156891273014901,
+    "coverage_at_error_budget": 0.693529411764706,
+    "mean_confidence": 0.9580137741914275,
+    "datasets": 17
+   }
+  },
+  "derived (Track B softmax of log-odds, same 17 datasets)": {
+   "ines": {
+    "brier": 0.5266089613872135,
+    "nll": 0.8626628971730973,
+    "ece": 0.11876053221277025,
+    "coverage_at_error_budget": 0.1335294117647059,
+    "mean_confidence": 0.4924301710578631,
+    "datasets": 17
+   },
+   "julia": {
+    "brier": 0.7685502933200778,
+    "nll": 1.5719702226511314,
+    "ece": 0.31343832319933446,
+    "coverage_at_error_budget": 0.014117647058823528,
+    "mean_confidence": 0.7770917395785171,
+    "datasets": 17
+   },
+   "laya": {
+    "brier": 0.35941673886508035,
+    "nll": 0.6807608663005745,
+    "ece": 0.12656927310814167,
+    "coverage_at_error_budget": 0.38176470588235295,
+    "mean_confidence": 0.7694262437653563,
+    "datasets": 17
+   },
+   "gliner": {
+    "brier": 0.5872442915941616,
+    "nll": 1.0308428769681912,
+    "ece": 0.2029575594285007,
+    "coverage_at_error_budget": 0.03705882352941177,
+    "mean_confidence": 0.7019368747032854,
+    "datasets": 17
+   }
+  },
+  "derived (Track B softmax of log-odds, all 22 datasets)": {
+   "ines": {
+    "brier": 0.6231953131399431,
+    "nll": 1.4522999819886815,
+    "ece": 0.12354853986128157,
+    "coverage_at_error_budget": 0.10590909090909091,
+    "mean_confidence": 0.39202771650296064,
+    "datasets": 22
+   },
+   "julia": {
+    "brier": 0.8554517402951244,
+    "nll": 2.468274766821923,
+    "ece": 0.31679964418123413,
+    "coverage_at_error_budget": 0.011363636363636364,
+    "mean_confidence": 0.687741939307485,
+    "datasets": 22
+   },
+   "laya": {
+    "brier": 0.47884708479764443,
+    "nll": 1.203588850522752,
+    "ece": 0.12330875423821963,
+    "coverage_at_error_budget": 0.29681818181818187,
+    "mean_confidence": 0.6690071844208508,
+    "datasets": 22
+   },
+   "gliner": {
+    "brier": 0.6643794367941797,
+    "nll": 1.5673671850720794,
+    "ece": 0.18732672413963056,
+    "coverage_at_error_budget": 0.031363636363636364,
+    "mean_confidence": 0.5655688289154754,
+    "datasets": 22
+   }
+  }
+ },
+ "ines_ablation": {
+  "agnews": {
+   "labels": 4,
+   "accuracy": {
+    "native(<=26)": 0.59,
+    "knockout-20": null,
+    "knockout-2": 0.64,
+    "one-vs-rest": 0.42
+   },
+   "macro_f1": {
+    "native(<=26)": 0.5843451916892106,
+    "knockout-20": null,
+    "knockout-2": 0.639691081949832,
+    "one-vs-rest": 0.32652938961350175
+   }
+  },
+  "emotiondair": {
+   "labels": 6,
+   "accuracy": {
+    "native(<=26)": 0.43,
+    "knockout-20": null,
+    "knockout-2": 0.43,
+    "one-vs-rest": 0.34
+   },
+   "macro_f1": {
+    "native(<=26)": 0.39042230395613853,
+    "knockout-20": null,
+    "knockout-2": 0.39217171717171717,
+    "one-vs-rest": 0.29478472669962036
+   }
+  },
+  "banking77": {
+   "labels": 72,
+   "accuracy": {
+    "native(<=26)": null,
+    "knockout-20": 0.39,
+    "knockout-2": 0.45,
+    "one-vs-rest": 0.29
+   },
+   "macro_f1": {
+    "native(<=26)": null,
+    "knockout-20": 0.34265873015873016,
+    "knockout-2": 0.4010912698412698,
+    "one-vs-rest": 0.2234898589065256
+   }
+  },
+  "financialphrasebank": {
+   "labels": 3,
+   "accuracy": {
+    "native(<=26)": 0.51,
+    "knockout-20": null,
+    "knockout-2": 0.44,
+    "one-vs-rest": 0.45
+   },
+   "macro_f1": {
+    "native(<=26)": 0.48911853494169083,
+    "knockout-20": null,
+    "knockout-2": 0.3744570258331727,
+    "one-vs-rest": 0.388558378810103
+   }
+  },
+  "empathetic": {
+   "labels": 32,
+   "accuracy": {
+    "native(<=26)": null,
+    "knockout-20": 0.25,
+    "knockout-2": 0.16,
+    "one-vs-rest": 0.18
+   },
+   "macro_f1": {
+    "native(<=26)": null,
+    "knockout-20": 0.1909003551273288,
+    "knockout-2": 0.10230728593731689,
+    "one-vs-rest": 0.14416745535166586
+   }
+  },
+  "massive": {
+   "labels": 59,
+   "accuracy": {
+    "native(<=26)": null,
+    "knockout-20": 0.35,
+    "knockout-2": 0.41,
+    "one-vs-rest": 0.2
+   },
+   "macro_f1": {
+    "native(<=26)": null,
+    "knockout-20": 0.2502433536331841,
+    "knockout-2": 0.3015873015873015,
+    "one-vs-rest": 0.15282852740479855
+   }
+  },
+  "yahootopics": {
+   "labels": 10,
+   "accuracy": {
+    "native(<=26)": 0.5,
+    "knockout-20": null,
+    "knockout-2": 0.57,
+    "one-vs-rest": 0.18
+   },
+   "macro_f1": {
+    "native(<=26)": 0.4915120854091442,
+    "knockout-20": null,
+    "knockout-2": 0.5584408865987813,
+    "one-vs-rest": 0.12307189542483658
+   }
+  },
+  "manifesto": {
+   "labels": 56,
+   "accuracy": {
+    "native(<=26)": null,
+    "knockout-20": 0.11,
+    "knockout-2": 0.18,
+    "one-vs-rest": 0.05
+   },
+   "macro_f1": {
+    "native(<=26)": null,
+    "knockout-20": 0.07504578754578754,
+    "knockout-2": 0.14917929292929294,
+    "one-vs-rest": 0.032902735562310034
+   }
+  },
+  "capsotu": {
+   "labels": 21,
+   "accuracy": {
+    "native(<=26)": 0.36,
+    "knockout-20": 0.37,
+    "knockout-2": 0.39,
+    "one-vs-rest": 0.21
+   },
+   "macro_f1": {
+    "native(<=26)": 0.3120540695728665,
+    "knockout-20": 0.3317682178584434,
+    "knockout-2": 0.3739835503810659,
+    "one-vs-rest": 0.17844985702128557
+   }
+  }
+ },
+ "anchors": {
+  "GLiNER2.5 Track A vs published pilot (accuracy)": {
+   "agnews": {
+    "ours": 0.7,
+    "published": 0.7
+   },
+   "emotiondair": {
+    "ours": 0.44,
+    "published": 0.44
+   },
+   "banking77": {
+    "ours": 0.61,
+    "published": 0.61
+   }
+  },
+  "Julia-1 Track A vs its model card (accuracy)": {
+   "agnews": {
+    "ours": 0.94,
+    "card": 0.94,
+    "ours_strategy": "single request"
+   },
+   "emotiondair": {
+    "ours": 0.86,
+    "card": 0.86,
+    "ours_strategy": "single request"
+   },
+   "banking77": {
+    "ours": 0.63,
+    "card": 0.64,
+    "ours_strategy": "Router(width=20, survivors=2)"
+   }
+  }
+ },
+ "ovr_ties": {
+  "ines": {
+   "examples_with_tie_at_max": 268,
+   "n": 2200
+  },
+  "julia": {
+   "examples_with_tie_at_max": 4,
+   "n": 2200
+  },
+  "laya": {
+   "examples_with_tie_at_max": 11,
+   "n": 2200
+  },
+  "gliner": {
+   "examples_with_tie_at_max": 0,
+   "n": 2200
+  }
+ },
+ "addon": {
+  "date": "2026-10-06",
+  "model": "RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic (DiffusionGemma-26B-A4B-it, FP8) via razorback16/openjev:0.4.0, model openjev-0.1, think 0",
+  "scope": "Track A (native) only; added after the btzsc22-ext-v1 freeze; the four original models are recomputed by the same code and match results.json exactly"
+ }
+}

eval/speed.md

@@ -34,3 +34,36 @@
   ~180k prompt tokens/s here, close to its isolated prefill ceiling (~230k).
 - Not a general speed ranking: different serving stacks (1 vs 4 processes, graphs vs none), different prompt
   lengths for the same request, one GPU.
+
+## Run 3 (2026-10-06): four models, one GPU, with memory
+
+Same load generator (`scripts/bench_jev.py`, closed loop, 16 client processes, HTTP/1.1 without keep-alive), same two
+request sets and levels as above, **one free H200 per model in turn** (GPU 2 of the node, nothing else on it). The
+only change to the load generator: the request's `model` field is taken from `BENCH_MODEL` (OpenJev rejects requests
+without a known model; the other servers ignore it). GPU memory was sampled every 0.5 s with `nvidia-smi` (whole GPU).
+Raw output: `speed_runs/2026-10-06_part1_gemma_laya.txt`, `speed_runs/2026-10-06_part2_gliner_ines.txt` (the first
+GLiNER and Ines-1 start attempts failed on environment issues before any request and were rerun in part 2).
+
+| model (params total / active) | server | short c=1 p50 / p95 | short max q/s (c) | typed c=1 p50 / p95 | typed max q/s (c) | GPU memory idle / peak |
+|---|---|---:|---:|---:|---:|---:|
+| **Ines-1** (1.59B / 405M, bf16) | `scripts/serve_jev.py`, 1 process, CUDA graphs, dynamic batching | **9.4 / 9.7 ms** | **1,896** (128) | 30.6 / **40.5 ms** | 544 (32) | 9.7 / 21.9 GB |
+| laya-multilingual arch. (322M, bf16) | production Jev server, 4 processes, micro-batching | 13.0 / 25.9 ms | 1,848 (256) | **20.6** / 89.2 ms | **840** (128) | 8.0 / 24.1 GB |
+| GLiNER2.5-multi arch. (287M) | GLiNER Jev server, 4 workers, **no batching** | 48.8 / 49.3 ms | 68 (64) | 123.1 / 129.3 ms | 49 (32) | 6.7 / 26.5 GB |
+| DiffusionGemma-26B-A4B-it FP8 (~26B / ~4B) | OpenJev 0.4.0 (vLLM), max 64 sequences | 91.7 / 94.0 ms | 79 (32) | 158.8 / 170.2 ms | 143 (32) | weights 25.8 GiB; **120.6 / 135.1 GB reserved** |
+
+Latency at the best-throughput level (p50 / p95): Ines-1 118 / 197 ms (short), 282 / 386 ms (typed); laya 212 / 483 ms,
+604 / 913 ms; GLiNER 1,729 / 3,805 ms, 2,070 / 6,871 ms; Gemma 718 / 1,439 ms, 1,008 / 1,545 ms.
+
+Reading:
+- **Ines-1 has the lowest single-request latency on short requests and matches the 322M encoder's peak short
+  throughput** with one process. On ~300-token questions the encoder is still faster (840 vs 544 questions/s,
+  20.6 vs 30.6 ms median), but Ines-1's tail is tighter (p95 40.5 vs 89.2 ms), as in runs 1–2.
+- **Gemma is 5–10× slower at c=1 and ~4–24× lower in peak throughput** than Ines-1 or laya, and returned HTTP 503 on
+  12 of the ~12,000 short requests sent at c ≥ 32 (none on typed cases). Its memory figure is vLLM's preallocation (`gpu_memory_utilization`
+  0.80: 25.8 GiB of weights + 83 GiB of KV cache), not a minimum; it needs at least the ~26 GiB of weights.
+- **GLiNER's numbers measure its server, not the architecture's ceiling**: that server answers one request per
+  worker with no batching, so throughput stays flat as concurrency rises. Input tokens per question are counted
+  differently by each server (`usage`): Ines-1 61.5 / 316.2, laya 35.0 / 311.6, GLiNER 8.5 / 37.0, Gemma 67.0 / 136.7.
+- Peak memory includes activations at the highest concurrency; idle is after load and warm-up. The encoder rows use
+  domain fine-tunes of those architectures (speed is the architecture's), as in runs 1–2.
+- Not a general speed ranking: different serving stacks, prompt templates and process counts, one GPU.

eval/speed_runs/2026-10-06_part1_gemma_laya.txt

@@ -0,0 +1,40 @@
+===== gemma 11:23:26
+VRAM_IDLE_MiB 120557
+== short (2 questions)
+  c=1    n=3000       11 req/s      22 questions/s   p50   91.7 ms  p90   93.1  p95   94.0  p99   95.9   input tok/question   67.0   rows/forward   0.0  in forward   0 %  {200: 3000}
+  c=8    n=3000       32 req/s      64 questions/s   p50  266.5 ms  p90  292.5  p95  300.6  p99  357.7   input tok/question   67.0   rows/forward   0.0  in forward   0 %  {200: 3000}
+  c=32   n=2992       40 req/s      79 questions/s   p50  718.2 ms  p90 1260.2  p95 1438.8  p99 1856.4   input tok/question   67.0   rows/forward   0.0  in forward   0 %  {200: 2985, 503: 7}
+  c=64   n=2992       38 req/s      76 questions/s   p50 1506.4 ms  p90 2498.3  p95 2818.2  p99 3528.7   input tok/question   67.0   rows/forward   0.0  in forward   0 %  {200: 2991, 503: 1}
+  c=128  n=2992       36 req/s      71 questions/s   p50 3376.2 ms  p90 4356.4  p95 4636.1  p99 5255.2   input tok/question   67.0   rows/forward   0.0  in forward   0 %  {200: 2988, 503: 4}
+  c=256  n=2992       36 req/s      72 questions/s   p50 6448.7 ms  p90 7529.1  p95 7880.5  p99 8527.4   input tok/question   67.0   rows/forward   0.0  in forward   0 %  {200: 2992}
+== typed-decisions ES (5 questions per case)
+  c=1    n=800         6 req/s      32 questions/s   p50  158.8 ms  p90  166.4  p95  170.2  p99  177.3   input tok/question  136.7   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=8    n=800        21 req/s     107 questions/s   p50  382.0 ms  p90  410.2  p95  429.2  p99  522.9   input tok/question  136.7   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=32   n=800        29 req/s     143 questions/s   p50 1007.7 ms  p90 1382.2  p95 1544.9  p99 1773.0   input tok/question  136.7   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=128  n=800        25 req/s     127 questions/s   p50 4236.0 ms  p90 4894.6  p95 5139.9  p99 5471.4   input tok/question  136.7   rows/forward   0.0  in forward   0 %  {200: 800}
+VRAM_PEAK_MiB 135149
+(APIServer pid=560) INFO 10-06 11:07:52 [api_utils.py:286] non-default args: {'model_tag': 'RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic', 'enable_auto_tool_choice': True, 'tool_call_parser': 'gemma4', 'host': '127.0.0.1', 'model': 'RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic', 'max_model_len': 65536, 'max_logprobs': 32, 'served_model_name': ['dgemma'], 'override_generation_config': {'max_new_tokens': None}, 'attention_backend': 'TRITON_ATTN', 'reasoning_parser': 'gemma4', 'gpu_memory_utilization': 0.8, 'enable_prefix_caching': True, 'limit_mm_per_prompt': {'image': 8, 'video': 0}, 'max_num_seqs': 64, 'async_scheduling': True, 'diffusion_config': {'canvas_length': 64}}
+(EngineCore pid=1209) INFO 10-06 11:08:27 [default_loader.py:430] Loading weights took 16.33 seconds
+(EngineCore pid=1209) INFO 10-06 11:08:28 [model_runner.py:408] Model loading took 25.83 GiB memory and 20.123655 seconds
+(EngineCore pid=1209) INFO 10-06 11:08:28 [utils.py:320] Using LBNHC KV cache layout.
+(EngineCore pid=1209) INFO 10-06 11:08:50 [gpu_worker.py:674] Available KV cache memory: 83.17 GiB
+(EngineCore pid=1209) INFO 10-06 11:08:50 [gpu_worker.py:689] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.8000 is equivalent to --gpu-memory-utilization=0.7968 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.8032. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
+(EngineCore pid=1209) INFO 10-06 11:08:50 [kv_cache_utils.py:2432] GPU KV cache size: 1,191,787 tokens, Maximum concurrency for 65,536 tokens per request: 18.19x
+(EngineCore pid=1209) INFO 10-06 11:09:07 [gpu_worker.py:860] CUDA graph pool memory: 0.13 GiB (actual), 0.44 GiB (estimated), difference: 0.31 GiB (231.6%).
+===== laya 11:39:11
+VRAM_IDLE_MiB 7986
+== short (2 questions)
+  c=1    n=3000       61 req/s     122 questions/s   p50   13.0 ms  p90   25.0  p95   25.9  p99   29.0   input tok/question   35.0   rows/forward   0.0  in forward   0 %  {200: 3000}
+  c=8    n=3000      284 req/s     567 questions/s   p50   28.0 ms  p90   33.1  p95   34.9  p99  107.4   input tok/question   35.0   rows/forward   0.0  in forward   0 %  {200: 3000}
+  c=32   n=2992      598 req/s    1197 questions/s   p50   37.8 ms  p90  107.6  p95  142.1  p99  213.6   input tok/question   35.0   rows/forward   0.0  in forward   0 %  {200: 2992}
+  c=64   n=2992      765 req/s    1529 questions/s   p50   50.2 ms  p90  182.6  p95  220.8  p99  261.2   input tok/question   35.0   rows/forward   0.0  in forward   0 %  {200: 2992}
+  c=128  n=2992      873 req/s    1747 questions/s   p50  115.0 ms  p90  219.9  p95  239.6  p99  272.7   input tok/question   35.0   rows/forward   0.0  in forward   0 %  {200: 2992}
+  c=256  n=2992      924 req/s    1848 questions/s   p50  211.6 ms  p90  344.3  p95  483.3  p99  566.3   input tok/question   35.0   rows/forward   0.0  in forward   0 %  {200: 2992}
+== typed-decisions ES (5 questions per case)
+  c=1    n=800        30 req/s     152 questions/s   p50   20.6 ms  p90   85.2  p95   89.2  p99  101.3   input tok/question  311.6   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=8    n=800       107 req/s     533 questions/s   p50   33.1 ms  p90  174.0  p95  208.0  p99  270.4   input tok/question  311.6   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=32   n=800       150 req/s     750 questions/s   p50  178.9 ms  p90  351.6  p95  454.6  p99  652.9   input tok/question  311.6   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=128  n=800       168 req/s     840 questions/s   p50  603.9 ms  p90  873.1  p95  912.8  p99 1285.5   input tok/question  311.6   rows/forward   0.0  in forward   0 %  {200: 800}
+VRAM_PEAK_MiB 24076
+NO ARRANCA http://localhost:30043
+Error response from daemon: No such container: banco-gliner

eval/speed_runs/2026-10-06_part2_gliner_ines.txt

@@ -0,0 +1,31 @@
+===== gliner 11:57:45
+VRAM_IDLE_MiB 6650
+== short (2 questions)
+  c=1    n=3000       20 req/s      40 questions/s   p50   48.8 ms  p90   49.1  p95   49.3  p99   51.2   input tok/question    8.5   rows/forward   0.0  in forward   0 %  {200: 3000}
+  c=8    n=3000       21 req/s      43 questions/s   p50  419.9 ms  p90  487.7  p95  510.9  p99  555.9   input tok/question    8.5   rows/forward   0.0  in forward   0 %  {200: 3000}
+  c=32   n=2992       21 req/s      42 questions/s   p50 1672.7 ms  p90 2033.1  p95 2096.2  p99 2262.7   input tok/question    8.5   rows/forward   0.0  in forward   0 %  {200: 2992}
+  c=64   n=2992       34 req/s      68 questions/s   p50 1729.4 ms  p90 3542.9  p95 3804.8  p99 4448.3   input tok/question    8.5   rows/forward   0.0  in forward   0 %  {200: 2992}
+  c=128  n=2992       33 req/s      65 questions/s   p50 2407.2 ms  p90 9051.0  p95 9471.4  p99 9858.6   input tok/question    8.5   rows/forward   0.0  in forward   0 %  {200: 2992}
+  c=256  n=2992       26 req/s      53 questions/s   p50 5912.3 ms  p90 17059.3  p95 17745.1  p99 17961.1   input tok/question    8.5   rows/forward   0.0  in forward   0 %  {200: 2992}
+== typed-decisions ES (5 questions per case)
+  c=1    n=800         8 req/s      40 questions/s   p50  123.1 ms  p90  125.5  p95  129.3  p99  198.9   input tok/question   37.0   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=8    n=800         8 req/s      41 questions/s   p50  997.9 ms  p90 1161.4  p95 1297.9  p99 1472.9   input tok/question   37.0   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=32   n=800        10 req/s      49 questions/s   p50 2069.6 ms  p90 6033.7  p95 6870.5  p99 7986.3   input tok/question   37.0   rows/forward   0.0  in forward   0 %  {200: 800}
+  c=128  n=800         9 req/s      45 questions/s   p50 11465.9 ms  p90 17167.3  p95 19262.3  p99 20541.8   input tok/question   37.0   rows/forward   0.0  in forward   0 %  {200: 800}
+VRAM_PEAK_MiB 26544
+===== ines 12:16:37
+VRAM_IDLE_MiB 9747
+== short (2 questions)
+  c=1    n=3000      105 req/s     210 questions/s   p50    9.4 ms  p90    9.7  p95    9.7  p99    9.8   input tok/question   61.5   rows/forward   2.0  in forward  70 %  {200: 3000}
+  c=8    n=3000      443 req/s     887 questions/s   p50   17.8 ms  p90   18.0  p95   18.1  p99   18.4   input tok/question   61.5   rows/forward   8.0  in forward  98 %  {200: 3000}
+  c=32   n=2992      664 req/s    1328 questions/s   p50   45.2 ms  p90   47.2  p95   52.9  p99  143.1   input tok/question   61.5   rows/forward  30.2  in forward  95 %  {200: 2992}
+  c=64   n=2992      891 req/s    1782 questions/s   p50   66.2 ms  p90   69.4  p95   83.9  p99  142.0   input tok/question   61.5   rows/forward  41.6  in forward  96 %  {200: 2992}
+  c=128  n=2992      948 req/s    1896 questions/s   p50  117.8 ms  p90  153.1  p95  197.1  p99  198.6   input tok/question   61.5   rows/forward  49.9  in forward  95 %  {200: 2992}
+  c=256  n=2992      908 req/s    1816 questions/s   p50  224.6 ms  p90  307.5  p95  343.7  p99  353.6   input tok/question   61.5   rows/forward  53.2  in forward  86 %  {200: 2992}
+== typed-decisions ES (5 questions per case)
+  c=1    n=800        32 req/s     162 questions/s   p50   30.6 ms  p90   38.4  p95   40.5  p99   48.4   input tok/question  316.2   rows/forward   5.0  in forward  55 %  {200: 800}
+  c=8    n=800        93 req/s     465 questions/s   p50   85.4 ms  p90  110.2  p95  116.2  p99  124.0   input tok/question  316.2   rows/forward  17.6  in forward  99 %  {200: 800}
+  c=32   n=800       109 req/s     544 questions/s   p50  282.3 ms  p90  370.3  p95  385.5  p99  457.8   input tok/question  316.2   rows/forward  48.4  in forward  99 %  {200: 800}
+  c=128  n=800       106 req/s     528 questions/s   p50 1022.8 ms  p90 1304.1  p95 1306.1  p99 1308.2   input tok/question  316.2   rows/forward  54.0  in forward  98 %  {200: 800}
+VRAM_PEAK_MiB 21871
+FIN 12:18:17