BTZSC-22 external zero-shot classification evaluation (eval/btzsc22/) and README section (release commit 329aac4)

CHANGELOG_CLAIMS.md

@@ -29,3 +29,4 @@ Each is kept here with what replaced it and how it was found.
 | 22 | `pipeline_tag: text-classification` | No `pipeline_tag`: on the private staging repo the Hub attached a generic text-classification widget that no inference provider can run with this custom interface; descriptive tags kept | Hub metadata of the private repo |
 | 23 | Licence scope: weights/config/tokenizer Apache-2.0 and code MIT, other files unstated | Every file is assigned: Apache-2.0 also for documentation, `paper/`, training records, the model's own eval outputs and SHA256SUMS; MIT also for environment/build files; test-set gold labels and teacher distributions in `eval/` keep their datasets' licences | README, NOTICE, LICENSING_NOTES.md |
 | 24 | Paper as a draft, not in the repository | Technical report v1 (`paper/Ines-1.pdf`, LaTeX sources included); only change from the frozen draft: the date line now reads "Ines-1 Technical Report, October 2026" | `paper/` |
+| 25 | No evaluation on tasks absent from training | BTZSC-22 (extension of the jev-benchmarks protocol): 22 datasets, 15 absent from the decision fine-tuning mixture; Ines-1 lowest mean macro-F1 under the model-specific track, resolved only against GLiNER2.5; one-vs-rest does not help Ines-1, pairwise knockout (descriptive ablation) improves 6/9 multiclass datasets | `eval/btzsc22/` |

LICENSING_NOTES.md

@@ -7,7 +7,7 @@ below were read on 2026-10-02 from the official Hugging Face dataset cards (revi
 ## 1. Code — MIT
 
 The modelling code (`mini_v41/`), the decision reader (`mini_v41_jev/`) and the scripts (`scripts/`, `examples/`,
-`training/code/`) are released under the MIT License; the text is in [LICENSE-CODE](LICENSE-CODE)
+`training/code/`, `eval/btzsc22/code/`) are released under the MIT License; the text is in [LICENSE-CODE](LICENSE-CODE)
 (Copyright (c) 2026 Endikavi). The environment and build files (`Dockerfile`, `.dockerignore`, `requirements.txt`,
 `requirements.lock`, `.gitattributes`, `.gitignore`) are MIT as well.
 
@@ -30,7 +30,8 @@ material below. Facts considered:
 **Other files under Apache-2.0** (scope set on 2026-10-02, same LICENSE and NOTICE): the documentation (`README.md` and
 the other `.md` files, `paper/`, `tasksource_license_audit.csv`), the training records (`training/*.json`), the model's own
 evaluation outputs in `eval/` and `SHA256SUMS`. **Not relicensed:** the gold labels and teacher distributions of public
-test sets included in `eval/items/` and `eval/reference_rows.json` keep the licences of their datasets.
+test sets included in `eval/items/` and `eval/reference_rows.json`, and the BTZSC label texts and targets in
+`eval/btzsc22/`, keep the licences of their datasets.
 
 These notes do not conclude whether trained weights are a derivative work of their training data; the licence of the
 weights does not change the terms of that data.

NOTICE

@@ -7,12 +7,12 @@ Licensed under the Apache License, Version 2.0 (see LICENSE):
   - the training records (training/*.json), the model's own evaluation outputs in eval/, and SHA256SUMS.
 
 Licensed under the MIT License (see LICENSE-CODE):
-  - the code: mini_v41/, mini_v41_jev/, scripts/, examples/, training/code/;
+  - the code: mini_v41/, mini_v41_jev/, scripts/, examples/, training/code/, eval/btzsc22/code/;
   - the environment and build files: Dockerfile, .dockerignore, requirements.txt, requirements.lock,
     .gitattributes, .gitignore.
 
 Not relicensed: the gold labels and teacher distributions of public test sets included in eval/items/ and
-eval/reference_rows.json keep the licences of their datasets.
+eval/reference_rows.json, and the BTZSC label texts and targets in eval/btzsc22/, keep the licences of their datasets.
 
 The model was trained on third-party data, which is not redistributed here and remains under its own terms.
 Attribution and provenance: THIRD_PARTY_DATA_NOTICE.md. In particular:

README.md

@@ -36,10 +36,11 @@ datasets:
 >   `tokenizer/`; the documentation (`README.md` and the other `.md` files, `paper/`, `tasksource_license_audit.csv`); the
 >   training records (`training/*.json`); the model's own evaluation outputs in `eval/`; `SHA256SUMS`.
 > - **MIT** ([LICENSE-CODE](LICENSE-CODE)): the code — `mini_v41/`, `mini_v41_jev/`, `scripts/`, `examples/`,
->   `training/code/` — and the environment and build files: `Dockerfile`, `.dockerignore`, `requirements.txt`,
+>   `training/code/`, `eval/btzsc22/code/` — and the environment and build files: `Dockerfile`, `.dockerignore`, `requirements.txt`,
 >   `requirements.lock`, `.gitattributes`, `.gitignore`.
 > - **Third-party terms, not relicensed:** the gold labels and teacher distributions of the public test sets that
->   `eval/items/` and `eval/reference_rows.json` include keep their datasets' licences. The training data is not
+>   `eval/items/`, `eval/reference_rows.json` and `eval/btzsc22/` (BTZSC label texts and targets) include keep their
+>   datasets' licences. The training data is not
 >   redistributed and remains under its own terms, including attribution, share-alike and copyleft ones: see
 >   [THIRD_PARTY_DATA_NOTICE.md](THIRD_PARTY_DATA_NOTICE.md) and [LICENSING_NOTES.md](LICENSING_NOTES.md).
 
@@ -166,7 +167,7 @@ generalisation.**
 
 - **tasksource held-out** (2,206 questions, 350 sources): **new examples of tasks seen in training.** Every source of
   this test is also in the Stage 2 mix (from its train split); no example group is shared. Accuracy 49.5 % [47.4–51.6]
-  (Stage 1: 38.5 %; paired p = 1e-14). It is not an unseen-task evaluation; there is none.
+  (Stage 1: 38.5 %; paired p = 1e-14). It is not an evaluation on tasks absent from training; for that, see BTZSC-22 below.
 - No typed-decisions test case, state, normalised state text or state structure appears in any training file.
 
 ### External figures, kept apart (no leaderboard ranking)
@@ -182,6 +183,30 @@ whole cases. Specialists trained on typed-decisions `train` have a separate tabl
 self-reported, 0.483 to 0.801); the card also notes that scores above its teacher self-agreement (0.735) indicate
 learning the teacher's quirks.
 
+### External zero-shot classification: BTZSC-22
+
+We evaluated Ines-1 on 22 public BTZSC classification datasets (100 examples per dataset), using an extension of the
+jev-benchmarks protocol. Fifteen of the 22 datasets are absent from Ines-1's decision fine-tuning mixture.
+
+Under the model-specific classification track, mean macro-F1 across datasets was 0.474 for Ines-1, 0.525 for Julia-1,
+0.530 for laya-multilingual, and 0.537 for GLiNER2.5. Using a paired hierarchical bootstrap with datasets as the unit of
+inference, the difference relative to Ines-1 was resolved for GLiNER2.5 (+0.063, 95% CI +0.014 to +0.107), but not for
+Julia-1 (+0.051, 95% CI −0.026 to +0.126) or laya-multilingual (+0.056, 95% CI −0.040 to +0.163). On the 15 datasets
+absent from Ines-1's decision fine-tuning mixture, none of the pairwise differences was resolved.
+
+For datasets with at most 20 labels, native ECE was 0.127 for Ines-1, 0.298 for Julia-1, 0.093 for laya-multilingual,
+and 0.159 for GLiNER2.5.
+
+We also tested alternative decompositions of multiclass decisions. A common one-vs-rest protocol did not improve Ines-1
+(mean macro-F1 0.457 overall, and 0.489 → 0.283 on the 3–20-label subset). In contrast, a descriptive pairwise-knockout
+ablation improved Ines-1 on 6 of 9 multiclass datasets (accuracy). This suggests that Ines-1's current interface benefits
+more from comparing a small set of competing alternatives than from independently scoring each label as a yes/no
+decision. The knockout result is order-dependent and is reported as an ablation rather than a general classification
+protocol.
+
+Full protocol, manifests, prediction hashes, per-dataset results, calibration details, and reproducibility notes are
+available in [`eval/btzsc22/`](eval/btzsc22/).
+
 ### Private domain benchmark
 
 On an internal held-out domain benchmark (9 Spanish subtasks, 1,026 questions; not released), generic typed-decision
@@ -284,7 +309,9 @@ mix (no family above ~29 %):
   Prior); API batching does not change that the model sees one question at a time.
 - **Specialist exposure to the benchmark.** The typed-decisions results are in-distribution for a model trained on
   its train split; they are not comparable with zero-shot results.
-- **No unseen-task evaluation.** tasksource measures new examples of tasks seen in training.
+- **Limited evaluation on tasks absent from training.** tasksource measures new examples of tasks seen in training;
+  BTZSC-22 includes 15 datasets absent from the decision fine-tuning mixture (100 examples each; pretraining exposure
+  not excluded), where no difference against the three classifiers is resolved.
 - **Uneven zero-shot domain transfer** (0.558 macro balanced accuracy on a private domain), including a class-prior
   shift on yes/no questions over long texts after Stage 2.
 - **Specialisation forgetting.** Domain fine-tuning cost 6–9 points of typed-decisions accuracy.
@@ -327,7 +354,7 @@ sacrificing single-question performance.
 
 `model.safetensors` (bf16), `config.json` (configuration + provenance), `tokenizer/`, `mini_v41/` (modelling code),
 `mini_v41_jev/` (reading decisions, `Decider`), `scripts/` (HTTP server, load test), `examples/`, `training/` (recipe,
-manifests, data audit, scripts), `eval/` (reports, per-item outputs, speed protocol), [ENVIRONMENT.md](ENVIRONMENT.md),
+manifests, data audit, scripts), `eval/` (reports, per-item outputs, speed protocol, `btzsc22/` external classification evaluation), [ENVIRONMENT.md](ENVIRONMENT.md),
 `requirements.txt`, `requirements.lock`, `Dockerfile`, [CHANGELOG_CLAIMS.md](CHANGELOG_CLAIMS.md),
 [THIRD_PARTY_DATA_NOTICE.md](THIRD_PARTY_DATA_NOTICE.md), [LICENSING_NOTES.md](LICENSING_NOTES.md),
 [TASKSOURCE_LICENSE_AUDIT.md](TASKSOURCE_LICENSE_AUDIT.md) + `tasksource_license_audit.csv`,

SHA256SUMS

@@ -1,17 +1,41 @@
 97ad8982da911d928c3423f164a38f80608b98289ac4f4b3f292d90ae0ef2262  .dockerignore
 026b23285c5d5e2b117ee5065229bf984d6ae6fad6b21ee1f04c934b7cad1d57  .gitattributes
 862263fa1f46c20f0d1e4dac5ffcc75abd55c08211b2c3864c5f8764b9d87793  .gitignore
-39a6b4be19209aad5b756bfcc4895783ee78e3dc1d20418bfada7b9bd1147b4c  CHANGELOG_CLAIMS.md
+4463cc803842f7fe2406ce4e15039ff73842826d0f0c222f6857f73f8da7f54a  CHANGELOG_CLAIMS.md
 e4a1aeafc1f6c01f8fff69aef8a2518631428f3ec96db4a9a56d7bb296cce95f  Dockerfile
 5e98ea3d3045927158b597521fcf127ac5569e656e4e665f0c94a35e2a191980  ENVIRONMENT.md
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
 6a7ddcad077839a44697c005ea7e6f455caecaa8054412913108d6e21a1c9dbe  LICENSE-CODE
-d371047b330ef8dbeb22847ba1da6d2d3a0a8c37f74e5b5aa5164a566422fd02  LICENSING_NOTES.md
-b1e43b9261ff12e9040fa4b1a36f5b357324e31385640de5cabd0f652e236dc3  NOTICE
-8de965bdd1acc8af7bc75db6a28356241227897632a6c04ce4559ab0d66b3a30  README.md
+121b31536257bd7147d9c9b054f6fb4e2bb68734024415c081ac0b50c7cf48f1  LICENSING_NOTES.md
+1faada60db1ccfc073ae063fc474d9f75cba0c9d2683d03ad66b2074efeb166a  NOTICE
+d74ebe08f28c4f95dbf275fa9ff015c4f6ca7aacf59e3f65add1e15c8572de3c  README.md
 6ec8a2bb55f9331d77b9adbeae7fd71eeb5614a988cd21eee18c7bde775df5ba  TASKSOURCE_LICENSE_AUDIT.md
-058dc5687ddb161d4dbfcf1703ae059e78698714b4ce2ddf1783262c10b95142  THIRD_PARTY_DATA_NOTICE.md
-fbdcb006a93dfc3f384b4c6b713ed33215b72409bf5500b9d6c209cf889e4e38  config.json
+1a4df8f10bd30b01e6acc19ef2d935faee5485f0c25452ff64ba6c8598fabd97  THIRD_PARTY_DATA_NOTICE.md
+757003b29ec1d935f236c06ea1e39cf9362c43de358f8b3df74326f8bfdbd40d  config.json
+8f296d2c1dbe85d285bb45cd80d841e733f84168ff130daf825933917684993b  eval/btzsc22/PROTOCOL.md
+5cc7f2cd08668abe6ac5dd5455e61bc2315e556e26c45906150426810fd49455  eval/btzsc22/REPORT.md
+95a4d4c459940195f03c6b937d750cd8e678527c4e2acaee1f7f2f3a9adc512c  eval/btzsc22/btzsc_test_files.json
+bb7dbba94cc130a09c77226e51cb7b72e5228685fbf8c2f78522290f485eff1e  eval/btzsc22/code/README.md
+afac77fa429f2f37c45e6cf46d0201551e9c98ddf9c01e7ba0cf1ac1feca0ef7  eval/btzsc22/code/btzsc22.yaml
+324bf63e3dad253faebaaa8a7a208f046e5f6e53872e1cf8dc925b0b049db1f7  eval/btzsc22/code/make_report.py
+a9b670b4bdaa77f878c31617407a7a8255cc5152fcc4196917390862cb536bb4  eval/btzsc22/code/prepare.py
+6ff68021abdf7957601a327e8fe7e1bca2c1148bdec3778f4afe1f5eb7d8f5d3  eval/btzsc22/code/run_model.py
+3f3fb8320cf6547cc359d64548adcd409b7d7c337c41773ff048e4f1ab8f6f15  eval/btzsc22/code/run_track.py
+12402f588a6ef7d0ca6202e6184330ace0de7474b9e529d0d0517f08c194c89a  eval/btzsc22/code/run_tracks.sh
+49d9f9abe284e3571251c655607ab3b3f49a0896febb92cbf3733af57a6803ad  eval/btzsc22/code/score_final.py
+8a2db365220f8fe59663fadfa6698c9117d38bf00a2a44beb462feead72af573  eval/btzsc22/manifest_ids.jsonl
+00da1891b1700fa327716f4c7e53d8c2ce1a23d3a20c5712972fdcc07db7952e  eval/btzsc22/per_dataset.csv
+2862d2a7c65571586b7b8b607fe0daafe758c37fcccd51e00c2db7fa84bf5e4c  eval/btzsc22/predictions/ablation_ines.knockout_2.jsonl
+e715c918b8a3b94a09661f51dad9df07c23d1b1dd468648a107cf435b408dd7d  eval/btzsc22/predictions/ablation_ines.knockout_20.jsonl
+b24c521b251795d3ec90d4d31fe0b2aa15856248afe588243207b8ef1ed98a1e  eval/btzsc22/predictions/trackA_native.gliner.jsonl
+673bb5373d0010481dbfc9f911cd85bebfed877d6ea759e2e3c0c7f10bb50873  eval/btzsc22/predictions/trackA_native.ines.jsonl
+914245b7b3545549998081b6df30e7b51f9ed725abffd36f9719ecf4044f548b  eval/btzsc22/predictions/trackA_native.julia.jsonl
+f65cd55dce20a876eb6ab366d6b5414528c302e357c71f3e1b14450a9eeab6bd  eval/btzsc22/predictions/trackA_native.laya.jsonl
+951aef27d70fed182d86381cb50bb7f0edb7c01c14fb9305fc2b67a423c5bba7  eval/btzsc22/predictions/trackB_ovr.gliner.jsonl
+102ddda9d71db407d6d519dd82dc691c76800584bfed7c5fd58dd12aa3559866  eval/btzsc22/predictions/trackB_ovr.ines.jsonl
+d0ca40988b8ad697223f7ccfc18ac35840c1390a5d46fabc219dab0c02069ed3  eval/btzsc22/predictions/trackB_ovr.julia.jsonl
+3c38c8a6e8afd8b80d0a66afa3fd8ae6c0f744e3f52b9cddc6d35d9c01c6ea4f  eval/btzsc22/predictions/trackB_ovr.laya.jsonl
+707f23cd6b4ae30a38573d4f29ce852a82878b23ee062c61a8025d4bae1360ff  eval/btzsc22/results.json
 da9bad7608ad2fd759e0e9088fd55a60a22d11ea6b8cb720040594a2b0fc556f  eval/fullcase_report.json
 60e288cf88dfdf189fa7582daaad8a274911ce47db6cf47f3dfe0367760fa353  eval/fullcase_report.md
 a8e81da13d548c4dac0d26ecab081c4640429f1493eb607015bbca8008000d29  eval/items/v3-engram_off__tasksource.jsonl
@@ -91,7 +115,7 @@ fb24a4241b3649948bcf4392ad7e553fbb271baa9bfb80d92260f590f7817977  training/code/
 156ddb49b6a8d3c7a4bb6369f5269359d5110f19b30087a578bbe6bbc29153d1  training/code/prepare_mix.py
 94f85491a03b57cb43693237813d461edc8be1363844f9bd295bce5ab28a8cbe  training/code/prepare_public.py
 c9fb2b03d023f961b9de00b4b5d458c0215c27d1a081f9ab708fa0ced664f5a8  training/code/reference_rows.py
-d06c63d039f7e10ebe4fe8c7f410a1f1219f89adff5c87ad3875fa52e8ed5dc2  training/code/release/build_release.py
+380e125fe024c921ebaf49b4ca55b36dbaa46b9c1087988dd7819a7a2b6fc8a7  training/code/release/build_release.py
 11448761b33a78703239d651e61d45f2e2ffe61c222ed5d9be15a33805f31398  training/code/train_decisions.py
 0c680ad5db1cd7867c64c50b742512897ab645d693cda4f66bb69eec84916b8c  training/data_audit.json
 79b091a8a4d542ba546254fe7123e4c458ac6a566a88f761d3c54f49eea49a1d  training/stage1_typed_jev.manifest.json

THIRD_PARTY_DATA_NOTICE.md

@@ -38,6 +38,10 @@ published.
 
 ## Evaluation-only data
 
+| Dataset / source | Project / author | URL / identifier | Licence / note | Used for |
+|---|---|---|---|---|
+| BTZSC (22 base datasets) | BTZSC authors (btzsc) | `btzsc/btzsc` @ `fef2a2ac` — https://huggingface.co/datasets/btzsc/btzsc | No licence in the HF metadata; the 22 underlying datasets keep their own terms. `eval/btzsc22/` contains example IDs, text hashes, BTZSC label texts and targets, not the example texts | External evaluation only (BTZSC-22), not training |
+
 The private domain benchmark mentioned in the model card is not part of this repository, and none of its data, labels
 or examples are included.
 

config.json

@@ -95,7 +95,7 @@
     "engram_backend_identity": "token_id:65536:65536:197f7a314b356f70",
     "training_code_commit": "0b5e63e3da05c1a51eb9f0ea1758f8d53d200a80-dirty",
     "bundle_model_code_commit": "5d1ad97",
-    "bundle_release_code_commit": "248b774",
+    "bundle_release_code_commit": "329aac4",
     "torch_version_at_training": "2.14.0+cu130"
   }
 }

eval/btzsc22/PROTOCOL.md

@@ -0,0 +1,202 @@
+# BTZSC-22 evaluation using an extension of the jev-benchmarks protocol
+
+> **Amendment (2026-10-02, after the run; documentation only — nothing executed was changed).** Track A is renamed from
+> "native" to **model-specific classification track**, because above their limits Julia-1 uses its Router and Ines-1 the
+> knockout fallback of this protocol. Added: BTZSC file hashes and manifest regeneration, full model revisions, the
+> Julia-1 Router build (Bend 2.0.27), the laya-multilingual option condition, Ines-1's > 26-label handling and the exact
+> definition of the paired hierarchical bootstrap. The original frozen text follows, with only the track name changed.
+
+Protocol revision `btzsc22-ext-v1`, frozen on 2026-10-02 **before** any Track A rerun, Track B run or final analysis.
+This is our own protocol. It extends the public jev-benchmarks pilot (`btzsc-pilot-v1`, frozen on AG News, DAIR
+Emotion and Banking77) to all 22 BTZSC base datasets; it is not the public pilot, and results on the 19 added datasets
+are not jev-benchmarks results.
+
+## Data
+
+| item | value |
+|---|---|
+| dataset | `btzsc/btzsc` @ `fef2a2ac62b69c58670047dddf045c53d7c3cb5e`, split `test`, BTZSC label descriptions (`hypothesis`) |
+| sampling | jev-benchmarks @ `0d610cc53e79bcbec691312b0c4adb4a0e371642`, `load_examples` unchanged: 100 examples per dataset, target-balanced, seed 20260917 + dataset position; rows with no single positive label excluded (Banking77 out-of-scope) |
+| datasets (order) | agnews, emotiondair, banking77, amazonpolarity, imdb, appreviews, yelpreviews, rottentomatoes, financialphrasebank, empathetic, biasframes_intent, massive, yahootopics, trueteacher, manifesto, capsotu, biasframes_offensive, biasframes_sex, wikitoxic_insult, wikitoxic_obscene, wikitoxic_threat, wikitoxic_toxicaggregated |
+| manifest | `manifest.jsonl`, 2,200 examples, sha256 `3edcd991c97fa3cf561367adf14351ea5b6c484fd317a4a2dab564d9ab2a37c8` (example IDs, text hashes, labels, targets) |
+| pilot identity | the first three datasets keep the pilot's order, so their 300 examples are the pilot's: rebuilding the pilot manifest with jev-benchmarks gives its published sha256 `ec064c52b149de458344cd4b4a44c158460f30b3bbb7fe8b2e7ec72d0abf3ba5`, and its IDs equal our first 300 |
+
+Label cardinality: 2 labels (13 datasets), 3–20 (agnews 4, emotiondair 6, financialphrasebank 3, yahootopics 10),
+over 20 (banking77 72, massive 59, manifesto 56, empathetic 32, capsotu 21). Public checkpoints may have seen these
+datasets; results measure benchmark performance, not uncontaminated generalisation.
+
+**Relation to Ines-1's training.** 15 datasets are **absent from Ines-1's decision fine-tuning mixture**. 7 belong to
+source families used in it (tasksource sources): amazonpolarity (`amazon_polarity`), banking77 (`banking77`), massive
+(`multilingual/massive`) and the four `wikitoxic_*` (`jigsaw_toxicity`, the same Jigsaw/Wikipedia comments). Nothing is
+claimed about pretraining, whose web corpora may contain any of them. The other models' training data is not known to us.
+
+## Models (weights hashed on the evaluation machine)
+
+| model | revision | weights sha256 | runtime |
+|---|---|---|---|
+| Ines-1 | `Endikavi/Ines-1` @ `965c51b8` | `7fa38ab3…c555` | `mini_v41_jev.Decider`, bf16 + autocast (canonical), one prompt per question |
+| Julia-1 | `SupersonicLabs/Julia-1` @ `a85b1273` | `df853bf7…5b72` | `julia.load_model(device="cuda", strict_encoding=True, max_length=8192, head_length=512)` (package `supersonic-julia` 0.1.0 from the model repo) |
+| laya-multilingual | `convaiinnovations/laya-multilingual` @ `e4e9ddf2` | `9d628fd9…f204` | `laya` 0.3.24, `laya.load(<local snapshot>)`, `predict(..., max_len=8192)` |
+| GLiNER2.5 | `fastino/gliner2.5-multi-v1` @ `235cf92d` | `c1ff4ec0…5395` | `gliner2` 2.0.0, `Classifier.from_pretrained(revision=…)` as in jev-benchmarks. gliner2 resolves only `config.json` at the pinned revision and the weights/tokenizer from `main` (`2ca71aaf`); `model.safetensors`, tokenizer and configs are byte-identical at both revisions (Hub file listings), only README/SKILL differ |
+
+Execution: one shared H200 (GPU also used by an unrelated training job: no latency is reported). PyTorch 2.14.0 in
+every environment. Julia-1 and laya run with `TORCHDYNAMO_DISABLE=1` and `TORCH_DISABLE_NATIVE_JIT=1` (PyTorch's eager
+Triton overrides need Python headers absent on the machine; aten kernels are used instead). GLiNER runs on GPU in fp32
+instead of the pilot's CPU: same argmax on 50/50 examples and max |Δp| 1.4e-5 against CPU.
+
+## Track A — model-specific classification track (frozen as "native")
+
+Each model receives the full label set through its normal, published interface. One request:
+`state = {"text": <text>}`, one Choice question `"Which single label best describes the input text?"`; GLiNER gets one
+exclusive classification schema over the label texts (as in jev-benchmarks).
+
+| model | ≤ limit | above its limit |
+|---|---|---|
+| Ines-1 | ≤ 26 labels (letters A–Z): one request, criteria `label_NNN → label text` (only the text is shown) | **no published interface.** Reported from the knockout ablation (groups of 20 in BTZSC order, each group's argmax advances), marked as such |
+| Julia-1 | ≤ 20: one request (`predict`), criteria `label_NNN → label text` | its documented `Router(engine, width=20, survivors=2)`: grouped narrowing and reranking; final probabilities cover the final candidates only (not a global distribution) |
+| laya-multilingual | no hard limit: one `predict` with criteria given **as the list of label texts** (laya renders a list label as itself, so no `label_NNN` appears; the runtime is unchanged). The README recommends ≤ ~20 options; its large-set path (`laya.shortlist.predict_shortlist`) needs an external embedding model and is **not** used |
+| GLiNER2.5 | no limit: one schema with all labels |
+
+## Track B — common one-vs-rest binary decomposition
+
+The same reduction for every model. For each label i, one independent yes/no question:
+`Does this text belong to the label "<label text>"?` — a `noul` question with no criteria for Ines-1, Julia-1 and laya;
+for GLiNER, an exclusive schema whose task name is that question and whose labels are `yes`, `no`.
+`s_i = log(P_yes / P_no)` with P clipped to [1e-12, 1 − 1e-12]; prediction `argmax_i s_i`, ties → lowest label index
+(ties counted and reported). `softmax(s)` is stored as a **derived multiclass probability**, never as the model's native
+probability. No temperature or threshold is fitted. The K questions are independent: Julia-1 and laya answer them in one
+batched call (checked equal to one-at-a-time on a made-up example, within 1e-4), Ines-1 one prompt each (canonical),
+GLiNER one call each.
+
+## Ablation (Ines-1)
+
+native (≤ 26) · knockout in groups of 20 · pairwise knockout (groups of 2, BTZSC order) · one-vs-rest (Track B). The two
+knockouts were run before this freeze with the exploratory runner `run_model.py` (same Choice request as Track A); they
+are reused, not rerun. Exploratory knockouts for the other models and laya runs with `label_NNN` keys are kept as records.
+
+## Metrics and statistics
+
+- Per dataset: accuracy and macro-F1 (`jev_benchmarks.metrics._macro_f1`); failures count as wrong.
+- Global: equal-weight mean of per-dataset macro-F1 over the 22 datasets; groups by cardinality and by fine-tuning
+  membership.
+- **Paired hierarchical bootstrap** (2,000 resamples, seed 20260917): resample the 22 datasets with replacement; within
+  each drawn dataset resample examples stratified by target; macro-F1 per dataset; equal-weight mean; paired difference
+  between two models on the same draws. 95 % percentile interval. The unit of inference is the dataset.
+- Probability quality, kept apart:
+  - *native* (Track A single requests): Brier, NLL, top-label ECE (10 bins), coverage at ≤ 5 % empirical error (jev-benchmarks `score_predictions`),
+    only on datasets where every model returns a full native distribution (≤ 20 labels, single request);
+  - *derived* (Track B, softmax of log-odds): the same metrics, labelled derived.
+- High-cardinality datasets (21–72 labels, 100 examples) give noisy per-dataset macro-F1: no per-dataset winner is claimed
+  there without a clear interval; they stay in the global mean.
+
+## Sanity anchors (reported with values)
+
+GLiNER2.5 Track A on the pilot's 300 examples vs the published pilot (accuracy 0.70 / 0.44 / 0.61); Julia-1 Track A vs
+its model card under shared conditions (AG News 94/100, DAIR Emotion 86/100, Banking77 64/100 with 1 abstention).
+
+## Frozen code
+
+`run_track.py` sha256 `3f3fb8320cf6547cc359d64548adcd409b7d7c337c41773ff048e4f1ab8f6f15` (templates are constants of
+that file); `score_final.py` hashed when written, before reading Track B results: sha256
+`49d9f9abe284e3571251c655607ab3b3f49a0896febb92cbf3733af57a6803ad` (recorded on the evaluation machine before the analysis ran).
+
+
+---
+
+## Amendment details (documentation only)
+
+### BTZSC source files
+
+`btzsc/btzsc` @ `fef2a2ac62b69c58670047dddf045c53d7c3cb5e`, one test Parquet per dataset; sha256 as published by the
+Hub (LFS oid) for that revision. Machine-readable copy: `btzsc_test_files.json`.
+
+| dataset | file | sha256 | bytes |
+|---|---|---|---:|
+| agnews | `agnews/test-00000-of-00001.parquet` | `b51a89341546018960b5724596ad3192f12d0649c5e45e91a06c0cee4253510f` | 1,280,949 |
+| emotiondair | `emotiondair/test-00000-of-00001.parquet` | `779970bb63817275a9d4004e4114ed00f2d86dc7670c36e13b4d018f85dc18ae` | 158,115 |
+| banking77 | `banking77/test-00000-of-00001.parquet` | `f42b14bddca9455891438e5e03ad2626e9de908eac0b65dbee4d5fa9eff8a776` | 804,682 |
+| amazonpolarity | `amazonpolarity/test-00000-of-00001.parquet` | `69dd2b714829e7aa44c15eb3e26e8773b16c1076b3f7de2dd5abde03c26783e9` | 2,974,010 |
+| imdb | `imdb/test-00000-of-00001.parquet` | `5f2e538b4c3fe71543ba661bdde6dd0c9b07e12cab5da3f170a1e254e61f237e` | 8,559,151 |
+| appreviews | `appreviews/test-00000-of-00001.parquet` | `085aaf8b64b36619f8e2b243a42d16c0959170b70627209b180fce89adc72a09` | 566,905 |
+| yelpreviews | `yelpreviews/test-00000-of-00001.parquet` | `f640df1fc45beccaa5f7c4a76419530bd6b7dc569200302469e006f34f738248` | 4,505,433 |
+| rottentomatoes | `rottentomatoes/test-00000-of-00001.parquet` | `e611f582e7357ea5a1ae5bd19e6eef0d7f7bbf233aad3c4c7e4800fbee163945` | 95,622 |
+| financialphrasebank | `financialphrasebank/test-00000-of-00001.parquet` | `af946f94e5dc2f5479ae8da33b70624a31f66e28e02d98d9426d399e6195401a` | 65,448 |
+| empathetic | `empathetic/test-00000-of-00001.parquet` | `c16923bdf05d2a2223ce9c8a696c8db1cfed83156e3b300eba5a547d6f42f078` | 1,092,730 |
+| biasframes_intent | `biasframes_intent/test-00000-of-00001.parquet` | `82a6da7eff047ac43fde1fdcc1d9b1b986cde08718d324f04af901607f585941` | 310,428 |
+| massive | `massive/test-00000-of-00001.parquet` | `ad109e13b5204ca4b80545d15d9f81167516b1e4ebd834f9d8e870039c9b8b96` | 558,077 |
+| yahootopics | `yahootopics/test-00000-of-00001.parquet` | `c51a3f3de236cc897f5328c39c0946455d82c7465ffb4d3b0ad52b83d56e9b50` | 19,108,728 |
+| trueteacher | `trueteacher/test-00000-of-00001.parquet` | `5e84c9d12755f2de3c40bf2e24b321f161db126bba12d0a6ba19b2da6587af46` | 6,972,936 |
+| manifesto | `manifesto/test-00000-of-00001.parquet` | `78169ff647e0218ef3959c7837a53c3e1e1b65731efdeb22752a1da31ae02a38` | 8,569,698 |
+| capsotu | `capsotu/test-00000-of-00001.parquet` | `33e1d9262a8923d35cb2e92c7b72f56a6caf2f55049165896e2b27d5934b39a2` | 723,183 |
+| biasframes_offensive | `biasframes_offensive/test-00000-of-00001.parquet` | `5b4df757e7241d8fd5456fe04545eea2bfc171b9462069ea369991c9b4cef10e` | 327,567 |
+| biasframes_sex | `biasframes_sex/test-00000-of-00001.parquet` | `7cb2a305f07378708941d792d858577eb552faded64958d91b6da3d7360c5057` | 379,857 |
+| wikitoxic_insult | `wikitoxic_insult/test-00000-of-00001.parquet` | `16701455c8c71451ba98a7ffc9b26af8d0a0a2d34da7c1b30e78d598cb60cda3` | 1,724,127 |
+| wikitoxic_obscene | `wikitoxic_obscene/test-00000-of-00001.parquet` | `733d7dd830d2fbb30c7b1f2261f68029f6889d2196349eb728350ea46a074298` | 1,847,410 |
+| wikitoxic_threat | `wikitoxic_threat/test-00000-of-00001.parquet` | `f088d3cd21c895a7d27cbacf901c1873cb5aca21576ff345d05c43f22ce0ac6a` | 1,332,140 |
+| wikitoxic_toxicaggregated | `wikitoxic_toxicaggregated/test-00000-of-00001.parquet` | `bc706516c9f218e57de4e418691fdfd956cb0baa34574811bfdb692bf28e1b5f` | 2,024,344 |
+
+### Regenerating the manifest
+
+```bash
+git clone https://github.com/AbdelStark/jev-benchmarks && git -C jev-benchmarks checkout 0d610cc53e79bcbec691312b0c4adb4a0e371642
+pip install -e "./jev-benchmarks[data]"            # datasets>=4,<5
+python code/prepare.py code/btzsc22.yaml manifest.jsonl
+sha256sum manifest.jsonl                            # 3edcd991c97fa3cf561367adf14351ea5b6c484fd317a4a2dab564d9ab2a37c8
+```
+
+`manifest.jsonl` contains the example texts and is not shipped; `manifest_ids.jsonl` is the same file without the `text`
+field (IDs, `text_sha256`, labels, targets), so a regenerated manifest can be checked row by row.
+
+### Exact model revisions
+
+| model | Hub repository @ revision | weights file sha256 |
+|---|---|---|
+| Ines-1 | `Endikavi/Ines-1` @ `965c51b8211a08a8d91c4f078eb7ffbf55e8d29e` | `model.safetensors` `7fa38ab3adf37530c5b150b0527d8cc9c24350b402cd2d8bf727e736b78ac555` |
+| Julia-1 | `SupersonicLabs/Julia-1` @ `a85b127321d580d65176c89ced8273f305745d85` | `model.safetensors` `df853bf7fe424420011f3d0c47a05d7341aa9eefa7fb9f203ea4aada4ad95b72` |
+| laya-multilingual | `convaiinnovations/laya-multilingual` @ `e4e9ddf21a7b1903b7acffd8814ad4307bf63a67` | `model.safetensors` `9d628fd971b700382ac6f65920a86f149777b2e748e0c955fb3b19695aa8f204` |
+| GLiNER2.5 | `fastino/gliner2.5-multi-v1` @ `235cf92d6d4318da9bfca0d08975c8fa7250d13b` (weights resolved from `main` = `2ca71aafb3446d9014e1c55c7ff51c9bc7209c47`, byte-identical files) | `model.safetensors` `c1ff4ec0bc00031c15530b8f3c33d3677f27949e6a0cb52e1247a6224b6c5395` |
+
+Runtimes: `mini_v41_jev` from the Ines-1 repository; `supersonic-julia` 0.1.0 (the Julia-1 repository, editable install);
+`laya` 0.3.24; `gliner2` 2.0.0; PyTorch 2.14.0 in every environment.
+
+### Julia-1 Router (Track A above 20 labels)
+
+`julia.router.Router(engine, width=20, survivors=2)` around the same engine as the single requests
+(`load_model(device="cuda", strict_encoding=True, max_length=8192, head_length=512)`). The Router loads a native
+reducer, `julia/router/build/libjulia_router.so`, which is not distributed with the model and is built by
+`julia/router/build.py` with Bend **2.0.27** and Clang:
+
+| item | value |
+|---|---|
+| Bend | official release `v2.0.27`, asset `bend-2.0.27-linux-x64.tar.gz`, sha256 `58adc86af6605ed0c48f7d84e4c23028f78893ce4a867a20a4f004b11582687b` (verified against the release digest) |
+| compiler | Ubuntu clang 18.1.3 (Ubuntu 24.04, glibc 2.39), portable build (no `--native-cpu`) |
+| `libjulia_router.so` | sha256 `79a346cc9166fa46d0a3a9583de1ca033e1299b8f38d61f4af2c207c58d95e96` |
+| check | the build script's own checks of the generated code passed; `python -m unittest discover -s julia/router/tests`: 30 tests OK |
+
+### laya-multilingual option condition
+
+Choice criteria are passed as a **list of label texts**, so laya renders each option as the label text itself; no
+`label_NNN` identifier reaches the model. This is laya's published API (`criteria` may be a list of labels) with the
+runtime unchanged. A discarded exploratory pilot used `label_NNN → text` dicts (rendered as `label_NNN: text`); see
+REPORT.md.
+
+### Ines-1 above 26 labels
+
+Ines-1 reads at most 26 options (letters A–Z) and has no published interface for more. In Track A, datasets with more
+than 26 labels (banking77 72, massive 59, manifesto 56, empathetic 32) use this protocol's fallback: labels split in
+BTZSC order into consecutive groups of at most 20; one Choice request per group; each group's argmax advances; repeated
+until one label remains. capsotu (21 labels) uses a single request.
+
+### Paired hierarchical bootstrap (as implemented in `code/score_final.py`)
+
+1. One generator, `numpy.random.default_rng(20260917)`, used sequentially for the groups in this order: all 22
+   (2,000 resamples), 2 labels, 3–20 labels, 21–72 labels, absent from the Ines-1 decision fine-tuning mixture,
+   families in that mixture (1,000 resamples each).
+2. Each resample draws n datasets with replacement from the group's n datasets (`integers(0, n, n)`).
+3. For each drawn dataset, in draw order, examples are resampled with replacement **within each target class** (same
+   class sizes; classes in increasing index), giving one index vector shared by every model and track.
+4. Macro-F1 is computed per model and track on that vector (`jev_benchmarks.metrics._macro_f1`, number of classes = the
+   dataset's label count; failed predictions count as wrong).
+5. For each pair, the difference (B − A) is averaged with equal weight over the drawn datasets (repeats count).
+6. The interval is the 2.5th and 97.5th percentiles (`numpy.quantile`, linear) of the resampled mean differences; the
+   point estimate is the mean per-dataset difference on the original data.

eval/btzsc22/REPORT.md

@@ -0,0 +1,159 @@
+# BTZSC-22 evaluation using an extension of the jev-benchmarks protocol — results
+
+**Main conclusion (this benchmark only).** Independent one-vs-rest binarization does not explain Ines-1's behavior.
+Ines-1 instead appears to benefit from reducing a multiclass decision to comparisons among a small number of competing
+alternatives. Under the model-specific classification track, Ines-1 has the lowest mean macro-F1 of the four models; the
+difference is resolved only against GLiNER2.5. Nothing here is generalised beyond these 22 datasets × 100 examples.
+
+**Tracks.** *Track A — model-specific classification track*: each model's own interface for the full label set, including
+Julia-1's Router above 20 labels and, for Ines-1 above 26 labels, the knockout fallback defined by this protocol (Ines-1
+has no published interface there). *Track B — common one-vs-rest*: the same binary decomposition for every model. In
+`results.json` and `per_dataset.csv`, keys `A` and `B` denote these two tracks.
+
+Protocol `btzsc22-ext-v1` ([PROTOCOL.md](PROTOCOL.md)). 22 BTZSC datasets × 100 examples (2,200; full manifest sha256 `3edcd991…37c8`, rebuilt by `code/prepare.py`; `manifest_ids.jsonl` here keeps IDs, text sha256, labels and targets but not the texts, which are not redistributed). Every number below comes from the frozen analysis script (`score_final.py`, sha256 `49d9f9ab…03ad`, hashed before Track B was read) over the saved predictions. Macro-F1 is per dataset; global values are equal-weight means over datasets. Intervals: paired hierarchical bootstrap (datasets resampled, then examples stratified by target), 95 %.
+
+## Sanity anchors
+
+| check | ours | reference |
+|---|---:|---:|
+| GLiNER2.5, agnews (pilot examples) | 70/100 | 70/100 (jev-benchmarks pilot) |
+| GLiNER2.5, emotiondair (pilot examples) | 44/100 | 44/100 (jev-benchmarks pilot) |
+| GLiNER2.5, banking77 (pilot examples) | 61/100 | 61/100 (jev-benchmarks pilot) |
+| Julia-1, agnews (single request) | 94/100 | 94/100 (model card) |
+| Julia-1, emotiondair (single request) | 86/100 | 86/100 (model card) |
+| Julia-1, banking77 (Router(width=20, survivors=2)) | 63/100 | 64/100 (model card, 1 abstention) |
+
+The pilot manifest rebuilt with jev-benchmarks has the published sha256 `ec064c52…3ba5` and its 300 IDs equal our first 300.
+
+## Track A — model-specific classification track
+
+| model | ≤ its limit | above its limit |
+|---|---|---|
+| Ines-1 | single request (≤ 26 labels) | knockout in groups of 20 (protocol fallback; banking77, empathetic, massive, manifesto) |
+| Julia-1 | single request (≤ 20 labels) | `Router(width=20, survivors=2)` (banking77, empathetic, massive, manifesto, capsotu) |
+| laya-multilingual | single request, all labels (label texts as options) | same |
+| GLiNER2.5 | single schema, all labels | same |
+
+Mean macro-F1 | Ines-1 | Julia-1 | laya-multilingual | GLiNER2.5
+|---|---:|---:|---:|---:|
+| all 22 | 0.474 | 0.525 | 0.530 | 0.537 |
+| 2 labels (13) | 0.562 | 0.583 | 0.722 | 0.627 |
+| 3-20 labels (4) | 0.489 | 0.634 | 0.559 | 0.541 |
+| 21-72 labels (5) | 0.234 | 0.287 | 0.008 | 0.297 |
+| absent from Ines-1 decision FT mixture (15) | 0.486 | 0.536 | 0.515 | 0.528 |
+| families in Ines-1 decision FT mixture (7) | 0.448 | 0.500 | 0.562 | 0.555 |
+
+Paired differences vs Ines-1 (other − Ines-1, mean macro-F1):
+
+| group | Julia-1 | laya-multilingual | GLiNER2.5 |
+|---|---:|---:|---:|
+| all 22 | +0.051 [-0.026, +0.126] | +0.056 [-0.040, +0.163] | +0.063 [+0.014, +0.107] |
+| 2 labels (13) | +0.021 [-0.061, +0.107] | +0.160 [+0.059, +0.272] | +0.066 [+0.005, +0.126] |
+| 3-20 labels (4) | +0.145 [-0.129, +0.426] | +0.070 [-0.230, +0.353] | +0.052 [-0.056, +0.160] |
+| 21-72 labels (5) | +0.053 [-0.043, +0.158] | -0.226 [-0.300, -0.122] | +0.063 [-0.039, +0.169] |
+| absent from Ines-1 decision FT mixture (15) | +0.050 [-0.029, +0.150] | +0.030 [-0.081, +0.144] | +0.042 [-0.007, +0.094] |
+| families in Ines-1 decision FT mixture (7) | +0.052 [-0.100, +0.178] | +0.113 [-0.125, +0.330] | +0.107 [+0.019, +0.191] |
+
+## Track B — common one-vs-rest binary decomposition
+
+Mean macro-F1 over the 22 datasets: Ines-1 0.457, Julia-1 0.357, laya-multilingual 0.629, GLiNER2.5 0.450. Paired
+intervals below.
+
+Mean macro-F1 | Ines-1 | Julia-1 | laya-multilingual | GLiNER2.5
+|---|---:|---:|---:|---:|
+| all 22 | 0.457 | 0.357 | 0.629 | 0.450 |
+| 2 labels (13) | 0.630 | 0.517 | 0.776 | 0.572 |
+| 3-20 labels (4) | 0.283 | 0.235 | 0.648 | 0.378 |
+| 21-72 labels (5) | 0.146 | 0.039 | 0.234 | 0.188 |
+| absent from Ines-1 decision FT mixture (15) | 0.409 | 0.332 | 0.611 | 0.428 |
+| families in Ines-1 decision FT mixture (7) | 0.558 | 0.411 | 0.668 | 0.495 |
+
+Paired differences vs Ines-1 (other − Ines-1, mean macro-F1):
+
+| group | Julia-1 | laya-multilingual | GLiNER2.5 |
+|---|---:|---:|---:|
+| all 22 | -0.100 [-0.146, -0.050] | +0.173 [+0.113, +0.237] | -0.007 [-0.070, +0.051] |
+| 2 labels (13) | -0.112 [-0.181, -0.043] | +0.146 [+0.095, +0.206] | -0.058 [-0.143, +0.030] |
+| 3-20 labels (4) | -0.048 [-0.162, +0.058] | +0.365 [+0.183, +0.554] | +0.095 [+0.019, +0.174] |
+| 21-72 labels (5) | -0.108 [-0.153, -0.049] | +0.088 [+0.011, +0.148] | +0.042 [-0.020, +0.111] |
+| absent from Ines-1 decision FT mixture (15) | -0.078 [-0.141, -0.014] | +0.202 [+0.127, +0.290] | +0.019 [-0.032, +0.072] |
+| families in Ines-1 decision FT mixture (7) | -0.147 [-0.199, -0.087] | +0.110 [+0.032, +0.194] | -0.064 [-0.219, +0.081] |
+
+## Effect of the binary decomposition within each model (Track B − Track A)
+
+For Ines-1 the one-vs-rest decomposition lowers mean macro-F1 on the 3–20-label subset (0.489 → 0.283, difference −0.206 [−0.316, −0.092]) and on the 21–72-label subset; over all 22 datasets the change is unresolved (−0.017 [−0.099, +0.070]).
+
+| group | Ines-1 | Julia-1 | laya-multilingual | GLiNER2.5 |
+|---|---:|---:|---:|---:|
+| all 22 | -0.017 [-0.099, +0.070] | -0.167 [-0.255, -0.088] | +0.099 [+0.045, +0.144] | -0.087 [-0.132, -0.041] |
+| 2 labels (13) | +0.068 [-0.049, +0.198] | -0.065 [-0.125, -0.013] | +0.054 [-0.001, +0.107] | -0.056 [-0.116, +0.008] |
+| 3-20 labels (4) | -0.206 [-0.316, -0.092] | -0.398 [-0.576, -0.208] | +0.089 [-0.009, +0.194] | -0.163 [-0.256, -0.072] |
+| 21-72 labels (5) | -0.088 [-0.126, -0.048] | -0.248 [-0.399, -0.099] | +0.226 [+0.128, +0.286] | -0.109 [-0.170, -0.032] |
+| absent from Ines-1 decision FT mixture (15) | -0.076 [-0.150, -0.009] | -0.204 [-0.300, -0.116] | +0.096 [+0.046, +0.144] | -0.100 [-0.146, -0.050] |
+| families in Ines-1 decision FT mixture (7) | +0.110 [-0.076, +0.293] | -0.089 [-0.245, +0.034] | +0.107 [-0.016, +0.208] | -0.061 [-0.150, +0.033] |
+
+## Ines-1 ablation (accuracy / macro-F1), datasets with more than 2 labels
+
+**Descriptive ablation, not the main protocol.** Pairwise knockout (two labels compared at a time, winners advancing in
+BTZSC label order) has higher accuracy than Ines-1's Track A result on 6 of the 9 multiclass datasets (agnews, banking77,
+massive, yahootopics, manifesto, capsotu), equal on 1 (emotiondair) and lower on 2 (financialphrasebank, empathetic). The
+result depends on the label order (bracket); no interval or test was computed and no statistical superiority is claimed.
+One-vs-rest has the lowest accuracy among the variants run on 7 of the 9 datasets (not on financialphrasebank or empathetic, where pairwise knockout is lower).
+
+| dataset | labels | native (≤ 26) | knockout groups of 20 | pairwise knockout | one-vs-rest |
+|---|---:|---|---|---|---|
+| agnews | 4 | 0.59 / 0.58 | – | 0.64 / 0.64 | 0.42 / 0.33 |
+| emotiondair | 6 | 0.43 / 0.39 | – | 0.43 / 0.39 | 0.34 / 0.29 |
+| banking77 | 72 | – | 0.39 / 0.34 | 0.45 / 0.40 | 0.29 / 0.22 |
+| financialphrasebank | 3 | 0.51 / 0.49 | – | 0.44 / 0.37 | 0.45 / 0.39 |
+| empathetic | 32 | – | 0.25 / 0.19 | 0.16 / 0.10 | 0.18 / 0.14 |
+| massive | 59 | – | 0.35 / 0.25 | 0.41 / 0.30 | 0.20 / 0.15 |
+| yahootopics | 10 | 0.50 / 0.49 | – | 0.57 / 0.56 | 0.18 / 0.12 |
+| manifesto | 56 | – | 0.11 / 0.08 | 0.18 / 0.15 | 0.05 / 0.03 |
+| capsotu | 21 | 0.36 / 0.31 | 0.37 / 0.33 | 0.39 / 0.37 | 0.21 / 0.18 |
+
+Knockouts were run with the exploratory runner before the freeze (same Choice request as Track A) and are reused, not rerun.
+
+## Probability quality (kept apart)
+
+**native (Track A, full native distributions, <=20 labels, 17 datasets)**
+
+| model | Brier | NLL | ECE | coverage at ≤ 5 % error | mean confidence |
+|---|---:|---:|---:|---:|---:|
+| Ines-1 | 0.501 | 0.800 | 0.127 | 0.089 | 0.612 |
+| Julia-1 | 0.645 | 1.662 | 0.298 | 0.129 | 0.894 |
+| laya-multilingual | 0.401 | 0.693 | 0.093 | 0.256 | 0.687 |
+| GLiNER2.5 | 0.485 | 0.860 | 0.159 | 0.197 | 0.731 |
+
+**derived (Track B softmax of log-odds, same 17 datasets)**
+
+| model | Brier | NLL | ECE | coverage at ≤ 5 % error | mean confidence |
+|---|---:|---:|---:|---:|---:|
+| Ines-1 | 0.527 | 0.863 | 0.119 | 0.134 | 0.492 |
+| Julia-1 | 0.769 | 1.572 | 0.313 | 0.014 | 0.777 |
+| laya-multilingual | 0.359 | 0.681 | 0.127 | 0.382 | 0.769 |
+| GLiNER2.5 | 0.587 | 1.031 | 0.203 | 0.037 | 0.702 |
+
+**derived (Track B softmax of log-odds, all 22 datasets)**
+
+| model | Brier | NLL | ECE | coverage at ≤ 5 % error | mean confidence |
+|---|---:|---:|---:|---:|---:|
+| Ines-1 | 0.623 | 1.452 | 0.124 | 0.106 | 0.392 |
+| Julia-1 | 0.855 | 2.468 | 0.317 | 0.011 | 0.688 |
+| laya-multilingual | 0.479 | 1.204 | 0.123 | 0.297 | 0.669 |
+| GLiNER2.5 | 0.664 | 1.567 | 0.187 | 0.031 | 0.566 |
+
+Derived probabilities are `softmax(log(P_yes/P_no))` over the K binary answers, not the models' native multiclass probabilities.
+
+On the 17 datasets with full native distributions, Ines-1 has lower native ECE than Julia-1 (0.127 vs 0.298) and
+GLiNER2.5 (0.159) and higher native ECE than laya-multilingual (0.093). Native and derived values are not compared with
+each other.
+
+## Notes
+
+- **One-vs-rest ties.** Deterministic rule, frozen before the run: prediction = argmax of `s_i = log(P_yes/P_no)` (P clipped to [1e-12, 1 − 1e-12]); when several labels share the maximum, the lowest label index in BTZSC order wins. Examples with a tie at the maximum: Ines-1 268/2200 (12 %; 95 of them correct), Julia-1 4, laya-multilingual 11, GLiNER2.5 0. Ines-1's ties are not saturation (no P_yes equals 0 or 1): its canonical bf16 readout gives P(yes) with coarse resolution, so different labels share the same value, mostly on high-cardinality datasets. They are reported as measured and not corrected post hoc.
+- **Discarded pilot (laya-multilingual).** A first exploratory run passed laya-multilingual the same `label_NNN → description` criteria as the other decision models; laya's runtime renders a dict criterion as `key: description`, so the IDs were shown to the model. Passing the label texts as a list instead — within laya's published API, runtime unchanged — raised its mean macro-F1 on the 17 datasets with ≤ 20 labels from 0.626 to 0.684. That pilot is discarded; all final results use the corrected protocol (label texts as options). Its predictions are not part of this package.
+- Julia-1's Router needs a native reducer built with Bend 2.0.27 + Clang (`julia/router/build.py`); built from the official Bend v2.0.27 linux-x64 release (sha256 verified) with Ubuntu clang 18.1.3; `libjulia_router.so` sha256 `79a346cc…5e96`; the Router's own 30 unit tests pass.
+- Other exploratory runs made before the freeze (knockouts for the other three models) are not part of the reported comparison.
+- High-cardinality datasets (21–72 labels) have 100 examples each; per-dataset macro-F1 there is noisy and no per-dataset winner is claimed.
+- The GPU was shared with an unrelated training job; no latency is reported.

eval/btzsc22/SHA256SUMS

@@ -0,0 +1,24 @@
+95a4d4c459940195f03c6b937d750cd8e678527c4e2acaee1f7f2f3a9adc512c  ./btzsc_test_files.json
+afac77fa429f2f37c45e6cf46d0201551e9c98ddf9c01e7ba0cf1ac1feca0ef7  ./code/btzsc22.yaml
+324bf63e3dad253faebaaa8a7a208f046e5f6e53872e1cf8dc925b0b049db1f7  ./code/make_report.py
+a9b670b4bdaa77f878c31617407a7a8255cc5152fcc4196917390862cb536bb4  ./code/prepare.py
+bb7dbba94cc130a09c77226e51cb7b72e5228685fbf8c2f78522290f485eff1e  ./code/README.md
+6ff68021abdf7957601a327e8fe7e1bca2c1148bdec3778f4afe1f5eb7d8f5d3  ./code/run_model.py
+3f3fb8320cf6547cc359d64548adcd409b7d7c337c41773ff048e4f1ab8f6f15  ./code/run_track.py
+12402f588a6ef7d0ca6202e6184330ace0de7474b9e529d0d0517f08c194c89a  ./code/run_tracks.sh
+49d9f9abe284e3571251c655607ab3b3f49a0896febb92cbf3733af57a6803ad  ./code/score_final.py
+8a2db365220f8fe59663fadfa6698c9117d38bf00a2a44beb462feead72af573  ./manifest_ids.jsonl
+00da1891b1700fa327716f4c7e53d8c2ce1a23d3a20c5712972fdcc07db7952e  ./per_dataset.csv
+e715c918b8a3b94a09661f51dad9df07c23d1b1dd468648a107cf435b408dd7d  ./predictions/ablation_ines.knockout_20.jsonl
+2862d2a7c65571586b7b8b607fe0daafe758c37fcccd51e00c2db7fa84bf5e4c  ./predictions/ablation_ines.knockout_2.jsonl
+b24c521b251795d3ec90d4d31fe0b2aa15856248afe588243207b8ef1ed98a1e  ./predictions/trackA_native.gliner.jsonl
+673bb5373d0010481dbfc9f911cd85bebfed877d6ea759e2e3c0c7f10bb50873  ./predictions/trackA_native.ines.jsonl
+914245b7b3545549998081b6df30e7b51f9ed725abffd36f9719ecf4044f548b  ./predictions/trackA_native.julia.jsonl
+f65cd55dce20a876eb6ab366d6b5414528c302e357c71f3e1b14450a9eeab6bd  ./predictions/trackA_native.laya.jsonl
+951aef27d70fed182d86381cb50bb7f0edb7c01c14fb9305fc2b67a423c5bba7  ./predictions/trackB_ovr.gliner.jsonl
+102ddda9d71db407d6d519dd82dc691c76800584bfed7c5fd58dd12aa3559866  ./predictions/trackB_ovr.ines.jsonl
+d0ca40988b8ad697223f7ccfc18ac35840c1390a5d46fabc219dab0c02069ed3  ./predictions/trackB_ovr.julia.jsonl
+3c38c8a6e8afd8b80d0a66afa3fd8ae6c0f744e3f52b9cddc6d35d9c01c6ea4f  ./predictions/trackB_ovr.laya.jsonl
+8f296d2c1dbe85d285bb45cd80d841e733f84168ff130daf825933917684993b  ./PROTOCOL.md
+5cc7f2cd08668abe6ac5dd5455e61bc2315e556e26c45906150426810fd49455  ./REPORT.md
+707f23cd6b4ae30a38573d4f29ce852a82878b23ee062c61a8025d4bae1360ff  ./results.json

eval/btzsc22/btzsc_test_files.json

@@ -0,0 +1,156 @@
+{
+ "agnews": [
+  [
+   "agnews/test-00000-of-00001.parquet",
+   "b51a89341546018960b5724596ad3192f12d0649c5e45e91a06c0cee4253510f",
+   1280949
+  ]
+ ],
+ "emotiondair": [
+  [
+   "emotiondair/test-00000-of-00001.parquet",
+   "779970bb63817275a9d4004e4114ed00f2d86dc7670c36e13b4d018f85dc18ae",
+   158115
+  ]
+ ],
+ "banking77": [
+  [
+   "banking77/test-00000-of-00001.parquet",
+   "f42b14bddca9455891438e5e03ad2626e9de908eac0b65dbee4d5fa9eff8a776",
+   804682
+  ]
+ ],
+ "amazonpolarity": [
+  [
+   "amazonpolarity/test-00000-of-00001.parquet",
+   "69dd2b714829e7aa44c15eb3e26e8773b16c1076b3f7de2dd5abde03c26783e9",
+   2974010
+  ]
+ ],
+ "imdb": [
+  [
+   "imdb/test-00000-of-00001.parquet",
+   "5f2e538b4c3fe71543ba661bdde6dd0c9b07e12cab5da3f170a1e254e61f237e",
+   8559151
+  ]
+ ],
+ "appreviews": [
+  [
+   "appreviews/test-00000-of-00001.parquet",
+   "085aaf8b64b36619f8e2b243a42d16c0959170b70627209b180fce89adc72a09",
+   566905
+  ]
+ ],
+ "yelpreviews": [
+  [
+   "yelpreviews/test-00000-of-00001.parquet",
+   "f640df1fc45beccaa5f7c4a76419530bd6b7dc569200302469e006f34f738248",
+   4505433
+  ]
+ ],
+ "rottentomatoes": [
+  [
+   "rottentomatoes/test-00000-of-00001.parquet",
+   "e611f582e7357ea5a1ae5bd19e6eef0d7f7bbf233aad3c4c7e4800fbee163945",
+   95622
+  ]
+ ],
+ "financialphrasebank": [
+  [
+   "financialphrasebank/test-00000-of-00001.parquet",
+   "af946f94e5dc2f5479ae8da33b70624a31f66e28e02d98d9426d399e6195401a",
+   65448
+  ]
+ ],
+ "empathetic": [
+  [
+   "empathetic/test-00000-of-00001.parquet",
+   "c16923bdf05d2a2223ce9c8a696c8db1cfed83156e3b300eba5a547d6f42f078",
+   1092730
+  ]
+ ],
+ "biasframes_intent": [
+  [
+   "biasframes_intent/test-00000-of-00001.parquet",
+   "82a6da7eff047ac43fde1fdcc1d9b1b986cde08718d324f04af901607f585941",
+   310428
+  ]
+ ],
+ "massive": [
+  [
+   "massive/test-00000-of-00001.parquet",
+   "ad109e13b5204ca4b80545d15d9f81167516b1e4ebd834f9d8e870039c9b8b96",
+   558077
+  ]
+ ],
+ "yahootopics": [
+  [
+   "yahootopics/test-00000-of-00001.parquet",
+   "c51a3f3de236cc897f5328c39c0946455d82c7465ffb4d3b0ad52b83d56e9b50",
+   19108728
+  ]
+ ],
+ "trueteacher": [
+  [
+   "trueteacher/test-00000-of-00001.parquet",
+   "5e84c9d12755f2de3c40bf2e24b321f161db126bba12d0a6ba19b2da6587af46",
+   6972936
+  ]
+ ],
+ "manifesto": [
+  [
+   "manifesto/test-00000-of-00001.parquet",
+   "78169ff647e0218ef3959c7837a53c3e1e1b65731efdeb22752a1da31ae02a38",
+   8569698
+  ]
+ ],
+ "capsotu": [
+  [
+   "capsotu/test-00000-of-00001.parquet",
+   "33e1d9262a8923d35cb2e92c7b72f56a6caf2f55049165896e2b27d5934b39a2",
+   723183
+  ]
+ ],
+ "biasframes_offensive": [
+  [
+   "biasframes_offensive/test-00000-of-00001.parquet",
+   "5b4df757e7241d8fd5456fe04545eea2bfc171b9462069ea369991c9b4cef10e",
+   327567
+  ]
+ ],
+ "biasframes_sex": [
+  [
+   "biasframes_sex/test-00000-of-00001.parquet",
+   "7cb2a305f07378708941d792d858577eb552faded64958d91b6da3d7360c5057",
+   379857
+  ]
+ ],
+ "wikitoxic_insult": [
+  [
+   "wikitoxic_insult/test-00000-of-00001.parquet",
+   "16701455c8c71451ba98a7ffc9b26af8d0a0a2d34da7c1b30e78d598cb60cda3",
+   1724127
+  ]
+ ],
+ "wikitoxic_obscene": [
+  [
+   "wikitoxic_obscene/test-00000-of-00001.parquet",
+   "733d7dd830d2fbb30c7b1f2261f68029f6889d2196349eb728350ea46a074298",
+   1847410
+  ]
+ ],
+ "wikitoxic_threat": [
+  [
+   "wikitoxic_threat/test-00000-of-00001.parquet",
+   "f088d3cd21c895a7d27cbacf901c1873cb5aca21576ff345d05c43f22ce0ac6a",
+   1332140
+  ]
+ ],
+ "wikitoxic_toxicaggregated": [
+  [
+   "wikitoxic_toxicaggregated/test-00000-of-00001.parquet",
+   "bc706516c9f218e57de4e418691fdfd956cb0baa34574811bfdb692bf28e1b5f",
+   2024344
+  ]
+ ]
+}

eval/btzsc22/code/README.md

@@ -0,0 +1,13 @@
+# Code of the BTZSC-22 evaluation
+
+| file | role | sha256 recorded in PROTOCOL.md |
+|---|---|---|
+| `btzsc22.yaml` | dataset list, seed and BTZSC revision (jev-benchmarks config format) | – |
+| `prepare.py` | builds `manifest.jsonl` with jev-benchmarks `load_examples`, unchanged | – |
+| `run_track.py` | Track A (model-specific classification) and Track B (one-vs-rest) for one model; templates are constants of this file | yes, frozen before the runs |
+| `run_tracks.sh` | runs both tracks for the four models (paths generalised, see header) | – |
+| `run_model.py` | exploratory runner used before the freeze; produced the Ines-1 knockout ablations in `../predictions/` | – |
+| `score_final.py` | the analysis (metrics, paired hierarchical bootstrap, calibration, ablation, anchors) | yes, frozen before Track B was read |
+| `make_report.py` | first draft of `../REPORT.md` and the package layout; the report text was then edited by hand | – |
+
+Only `btzsc22.yaml` (`output_dir`) and `run_tracks.sh` (machine paths) differ from the executed copies.

eval/btzsc22/code/btzsc22.yaml

@@ -0,0 +1,37 @@
+# Extension of jev-benchmarks pilot-v1 to all 22 BTZSC base datasets. The first three keep the pilot's order,
+# so their 100-example samples are exactly the pilot's (same seed + dataset offset).
+experiment_id: btzsc22-ines1-v1
+protocol_revision: pilot-v1-extended
+seed: 20260917
+dataset:
+  repository: btzsc/btzsc
+  revision: fef2a2ac62b69c58670047dddf045c53d7c3cb5e
+  samples_per_dataset: 100
+  datasets:
+    - {name: agnews, task: topic}
+    - {name: emotiondair, task: emotion}
+    - {name: banking77, task: intent}
+    - {name: amazonpolarity, task: sentiment}
+    - {name: imdb, task: sentiment}
+    - {name: appreviews, task: sentiment}
+    - {name: yelpreviews, task: sentiment}
+    - {name: rottentomatoes, task: sentiment}
+    - {name: financialphrasebank, task: sentiment}
+    - {name: empathetic, task: emotion}
+    - {name: biasframes_intent, task: intent}
+    - {name: massive, task: intent}
+    - {name: yahootopics, task: topic}
+    - {name: trueteacher, task: topic}
+    - {name: manifesto, task: topic}
+    - {name: capsotu, task: topic}
+    - {name: biasframes_offensive, task: topic}
+    - {name: biasframes_sex, task: topic}
+    - {name: wikitoxic_insult, task: topic}
+    - {name: wikitoxic_obscene, task: topic}
+    - {name: wikitoxic_threat, task: topic}
+    - {name: wikitoxic_toxicaggregated, task: topic}
+models:
+  gliner: {model_id: fastino/gliner2.5-multi-v1, revision: 235cf92d6d4318da9bfca0d08975c8fa7250d13b, device: cuda, batch_size: 1}
+  jev: {model_id: none, question: "Which single label best describes the input text?"}
+metrics: {ece_bins: 10, error_budget: 0.05, bootstrap_resamples: 2000}
+output_dir: runs/btzsc22

eval/btzsc22/code/make_report.py

@@ -0,0 +1,84 @@
+"""Assemble eval/btzsc22 (report tables + files). argv: btzsc22_dir out_dir"""
+import hashlib, json, shutil, sys
+from pathlib import Path
+B, O = Path(sys.argv[1]), Path(sys.argv[2]); O.mkdir(parents=True, exist_ok=True)
+R = json.load(open(B / "v1/analysis/results.json"))
+M = ["ines", "julia", "laya", "gliner"]; N = {"ines": "Ines-1", "julia": "Julia-1", "laya": "laya-multilingual", "gliner": "GLiNER2.5"}
+man = [json.loads(l) for l in open(B / "manifest.jsonl")]
+def mf1(t, p, k):
+    s = []
+    for c in range(k):
+        tp = sum(1 for a, b in zip(t, p) if a == c and b == c); fp = sum(1 for a, b in zip(t, p) if a != c and b == c); fn = sum(1 for a, b in zip(t, p) if a == c and b != c)
+        s.append(0 if 2 * tp + fp + fn == 0 else 2 * tp / (2 * tp + fp + fn))
+    return sum(s) / k
+# laya label_NNN (exploratory) vs label-text (v1) on the 17 datasets <= 20 labels, same single request otherwise
+old = {json.loads(l)["example_id"]: json.loads(l) for l in open(B / "laya.native.jsonl")}
+new = {json.loads(l)["example_id"]: json.loads(l) for l in open(B / "v1/laya.native.jsonl")}
+ds17 = [d for d, x in R["per_dataset"].items() if x["labels"] <= 20]
+def mean_f1(src):
+    v = []
+    for d in ds17:
+        es = [e for e in man if e["dataset"] == d]
+        v.append(mf1([e["target_index"] for e in es], [src[e["example_id"]]["predicted_index"] for e in es], len(es[0]["labels"])))
+    return sum(v) / len(v)
+asym = {"label_NNN keys (exploratory)": mean_f1(old), "label texts (v1, Track A)": mean_f1(new)}
+f = lambda x: "%.3f" % x
+ci = lambda v: "%+.3f [%+.3f, %+.3f]" % (v["difference"], *v["ci95"])
+L = []
+L.append("# BTZSC-22 evaluation using an extension of the jev-benchmarks protocol — results\n")
+L.append("Protocol `btzsc22-ext-v1` ([PROTOCOL.md](PROTOCOL.md)). 22 BTZSC datasets × 100 examples (2,200; manifest sha256 `3edcd991…37c8`). Every number below comes from the frozen analysis script (`score_final.py`, sha256 `49d9f9ab…03ad`, hashed before Track B was read) over the saved predictions. Macro-F1 is per dataset; global values are equal-weight means over datasets. Intervals: paired hierarchical bootstrap (datasets resampled, then examples stratified by target), 95 %.\n")
+L.append("## Sanity anchors\n")
+L.append("| check | ours | reference |\n|---|---:|---:|")
+for d, v in R["anchors"]["GLiNER2.5 Track A vs published pilot (accuracy)"].items():
+    L.append("| GLiNER2.5, %s (pilot examples) | %d/100 | %d/100 (jev-benchmarks pilot) |" % (d, round(v["ours"] * 100), round(v["published"] * 100)))
+for d, v in R["anchors"]["Julia-1 Track A vs its model card (accuracy)"].items():
+    L.append("| Julia-1, %s (%s) | %d/100 | %d/100 (model card%s) |" % (d, v["ours_strategy"], round(v["ours"] * 100), round(v["card"] * 100), ", 1 abstention" if d == "banking77" else ""))
+L.append("\nThe pilot manifest rebuilt with jev-benchmarks has the published sha256 `ec064c52…3ba5` and its 300 IDs equal our first 300.\n")
+for tr, title in (("A", "Track A — native interfaces"), ("B", "Track B — common one-vs-rest binary decomposition")):
+    L.append("## %s\n" % title)
+    L.append("Mean macro-F1 | " + " | ".join(N[m] for m in M) + "\n|---|" + "---:|" * 4)
+    for g, x in R["global"].items():
+        L.append("| %s | %s |" % (g, " | ".join(f(x[tr][m]) for m in M)))
+    L.append("\nPaired differences vs Ines-1 (other − Ines-1, mean macro-F1):\n")
+    L.append("| group | " + " | ".join(N[m] for m in M[1:]) + " |\n|---|---:|---:|---:|")
+    for g, b in R["bootstrap"].items():
+        L.append("| %s | %s |" % (g, " | ".join(ci(b["%s:%s - %s:ines" % (tr, m, tr)]) for m in M[1:])))
+    L.append("")
+L.append("## Effect of the binary decomposition within each model (Track B − Track A)\n")
+L.append("| group | " + " | ".join(N[m] for m in M) + " |\n|---|---:|---:|---:|---:|")
+for g, b in R["bootstrap"].items():
+    L.append("| %s | %s |" % (g, " | ".join(ci(b["B:%s - A:%s" % (m, m)]) for m in M)))
+L.append("\n## Ines-1 ablation (accuracy / macro-F1), datasets with more than 2 labels\n")
+L.append("| dataset | labels | native (≤ 26) | knockout groups of 20 | pairwise knockout | one-vs-rest |\n|---|---:|---|---|---|---|")
+g = lambda a, b: "–" if a is None else "%.2f / %.2f" % (a, b)
+for d, x in R["ines_ablation"].items():
+    a, m_ = x["accuracy"], x["macro_f1"]
+    L.append("| %s | %d | %s | %s | %s | %s |" % (d, x["labels"], g(a["native(<=26)"], m_["native(<=26)"]), g(a["knockout-20"], m_["knockout-20"]), g(a["knockout-2"], m_["knockout-2"]), g(a["one-vs-rest"], m_["one-vs-rest"])))
+L.append("\nKnockouts were run with the exploratory runner before the freeze (same Choice request as Track A) and are reused, not rerun.\n")
+L.append("## Probability quality (kept apart)\n")
+for k, c in R["calibration"].items():
+    L.append("**%s**\n\n| model | Brier | NLL | ECE | coverage at ≤ 5 %% error | mean confidence |\n|---|---:|---:|---:|---:|---:|" % k)
+    for m in M:
+        if m in c:
+            v = c[m]; L.append("| %s | %.3f | %.3f | %.3f | %.3f | %.3f |" % (N[m], v["brier"], v["nll"], v["ece"], v["coverage_at_error_budget"], v["mean_confidence"]))
+    L.append("")
+L.append("Derived probabilities are `softmax(log(P_yes/P_no))` over the K binary answers, not the models' native multiclass probabilities.\n")
+L.append("## Notes\n")
+L.append("- One-vs-rest ties at the maximum (lowest label index wins, as frozen): " + ", ".join("%s %d/2200" % (N[m], v["examples_with_tie_at_max"]) for m, v in R["ovr_ties"].items()) + ". Ines-1's ties are not saturation (no P = 0 or 1): its canonical bf16 readout gives P(yes) with coarse resolution, so different labels share the same value, mostly on high-cardinality datasets.")
+L.append("- laya-multilingual with `label_NNN` keys (exploratory runs, keys shown before each description) vs label texts as options (v1): mean macro-F1 on the 17 datasets with ≤ 20 labels %.3f → %.3f. The v1 Track A uses label texts; the asymmetry is removed." % (asym["label_NNN keys (exploratory)"], asym["label texts (v1, Track A)"]))
+L.append("- Julia-1's Router needs a native reducer built with Bend 2.0.27 + Clang (`julia/router/build.py`); built from the official Bend v2.0.27 linux-x64 release (sha256 verified) with Ubuntu clang 18.1.3; `libjulia_router.so` sha256 `79a346cc…5e96`; the Router's own 30 unit tests pass.")
+L.append("- High-cardinality datasets (21–72 labels) have 100 examples each; per-dataset macro-F1 there is noisy and no per-dataset winner is claimed.")
+L.append("- The GPU was shared with an unrelated training job; no latency is reported.")
+(O / "REPORT.md").write_text("\n".join(L) + "\n")
+for src, dst in (("manifest.jsonl", "manifest.jsonl"), ("v1/PROTOCOL.md", None), ("PROTOCOL.md", "PROTOCOL.md")):
+    p = B / src
+    if p.exists() and dst: shutil.copy(p, O / dst)
+(O / "predictions").mkdir(exist_ok=True)
+for m in M:
+    for t in ("native", "ovr"):
+        shutil.copy(B / ("v1/%s.%s.jsonl" % (m, t)), O / "predictions" / ("trackA_native.%s.jsonl" % m if t == "native" else "trackB_ovr.%s.jsonl" % m))
+for k in ("k20", "k2"):
+    shutil.copy(B / ("ines.%s.jsonl" % k), O / "predictions" / ("ablation_ines.knockout_%s.jsonl" % k[1:]))
+for x in ("results.json", "per_dataset.csv"):
+    shutil.copy(B / "v1/analysis" / x, O / x)
+print(json.dumps(asym))

eval/btzsc22/code/prepare.py

@@ -0,0 +1,12 @@
+"""Manifest of the BTZSC examples (jev-benchmarks load_examples, unchanged). argv: config out.jsonl"""
+import json, sys
+from jev_benchmarks.config import load_config
+from jev_benchmarks.data import load_examples
+cfg = load_config(sys.argv[1])
+ex = load_examples(cfg)
+with open(sys.argv[2], "w") as f:
+    for e in ex:
+        f.write(json.dumps(e.to_dict(), ensure_ascii=False) + "\n")
+import collections
+c = collections.Counter(e.dataset for e in ex); n = {e.dataset: len(e.labels) for e in ex}
+print(len(ex), {k: (c[k], n[k]) for k in c})

eval/btzsc22/code/run_model.py

@@ -0,0 +1,90 @@
+"""One backend over the manifest. argv: backend manifest out.jsonl mode [group]
+mode native: one Choice with all labels (skipped where the backend's limit is exceeded)
+mode knockout: groups of <= group labels in BTZSC order, each group's winner advances, until one remains
+Same request for every decision backend: state={"text": text}, one Choice question, options label_000...
+"""
+import json, os, sys, time
+BACKEND, MAN, OUT, MODE = sys.argv[1:5]
+G = int(sys.argv[5]) if len(sys.argv) > 5 else 0
+Q = "Which single label best describes the input text?"
+LIMIT = {"ines": 26, "julia": 20, "laya": 20, "gliner": 10 ** 9}[BACKEND]
+
+if BACKEND == "ines":
+    sys.path.insert(0, os.environ["INES"])
+    from mini_v41_jev import Decider
+    d = Decider(os.environ["INES"], device="cuda")
+    def call(text, labels):
+        crit = {"label_%03d" % i: l for i, l in enumerate(labels)}
+        a = d.decide({"text": text}, {"label": {"type": "choice", "instructions": Q, "criteria": crit}}, lang="en")["label"]
+        return [float(a["probabilities"][k]) for k in crit]
+elif BACKEND == "julia":
+    from julia import load_model
+    eng = load_model(os.environ["JULIA"], device="cuda", strict_encoding=True, max_length=8192, head_length=512)
+    def call(text, labels):
+        crit = {"label_%03d" % i: l for i, l in enumerate(labels)}
+        a = eng.predict(state={"text": text}, questions={"label": {"type": "choice", "instructions": Q, "criteria": crit}})["answers"]["label"]
+        return [float(a["probabilities"][k]) for k in crit]
+elif BACKEND == "laya":
+    import laya
+    ag = laya.load(os.environ.get("LAYA", "convaiinnovations/laya-multilingual"))
+    def call(text, labels):
+        crit = {"label_%03d" % i: l for i, l in enumerate(labels)}
+        a = ag.predict({"text": text}, {"label": {"type": "choice", "instructions": Q, "criteria": crit}}, max_len=8192)["answers"]["label"]
+        return [float(a["probabilities"][k]) for k in crit]
+elif BACKEND == "gliner":
+    from gliner2.classification import ClassificationSchema, Classifier
+    clf = Classifier.from_pretrained("fastino/gliner2.5-multi-v1", revision="235cf92d6d4318da9bfca0d08975c8fa7250d13b", device="cpu").eval()  # protocol: CPU
+    if os.environ.get("GLINER_DEVICE") == "cuda":  # checked: same argmax on 50/50 examples, max |dp| 1.4e-5 vs CPU
+        import torch
+        clf = clf.to("cuda")
+    cache = {}
+    def call(text, labels):
+        k = tuple(labels)
+        if k not in cache:
+            cache[k] = ClassificationSchema().single("label", list(labels))
+        r = clf.classify(text, cache[k])
+        return [float(r.probabilities("label")[l]) for l in labels]
+
+def knockout(text, labels, g):
+    alive = list(range(len(labels))); calls = 0
+    while len(alive) > 1:
+        nxt = []
+        for i in range(0, len(alive), g):
+            grp = alive[i:i + g]
+            if len(grp) == 1:
+                nxt.append(grp[0]); continue
+            p = call(text, [labels[j] for j in grp]); calls += 1
+            nxt.append(grp[max(range(len(grp)), key=p.__getitem__)])
+        alive = nxt
+    return alive[0], calls
+
+done = set()
+if os.path.exists(OUT):
+    done = {json.loads(l)["example_id"] for l in open(OUT)}
+ex = [json.loads(l) for l in open(MAN)]
+call(ex[0]["text"], ex[0]["labels"][:2])  # warm-up, not timed
+with open(OUT, "a") as f:
+    for e in ex:
+        if e["example_id"] in done:
+            continue
+        L = e["labels"]
+        if MODE == "native" and len(L) > LIMIT:
+            continue
+        if MODE == "knockout" and len(L) <= G and G != 2:
+            continue
+        if MODE == "knockout" and G == 2 and len(L) <= 2:
+            continue
+        t = time.perf_counter()
+        try:
+            if MODE == "native":
+                p = call(e["text"], L); pred = max(range(len(p)), key=p.__getitem__); calls = 1
+            else:
+                pred, calls = knockout(e["text"], L, G); p = None
+            err = None
+        except Exception as x:  # failures stay in the denominator
+            p, pred, calls, err = None, -1, 0, repr(x)[:300]
+        f.write(json.dumps({"backend": BACKEND, "mode": MODE, "group": G, "dataset": e["dataset"], "example_id": e["example_id"],
+                            "target_index": e["target_index"], "predicted_index": pred, "n_labels": len(L), "probabilities": p,
+                            "calls": calls, "latency_seconds": time.perf_counter() - t, "error": err}) + "\n")
+        f.flush()
+print("done", BACKEND, MODE, G)

eval/btzsc22/code/run_track.py

@@ -0,0 +1,107 @@
+"""BTZSC-22 extension, frozen tracks. argv: backend manifest out.jsonl track
+track native : the backend's published interface for the full label set (see PROTOCOL.md, Track A)
+track ovr    : common one-vs-rest binary decomposition, the same for every backend (Track B)
+Templates are constants of this file; PROTOCOL.md records its sha256.
+"""
+import json, math, os, sys, time
+BACKEND, MAN, OUT, TRACK = sys.argv[1:5]
+Q_CHOICE = "Which single label best describes the input text?"            # Track A, decision models
+Q_BINARY = 'Does this text belong to the label "{label}"?'                # Track B, every model
+EPS = 1e-12
+
+if BACKEND == "ines":
+    sys.path.insert(0, os.environ["INES"])
+    from mini_v41_jev import Decider
+    d = Decider(os.environ["INES"], device="cuda")
+    def choice(text, labels):                      # <= 26 options (letters A-Z)
+        crit = {"label_%03d" % i: l for i, l in enumerate(labels)}
+        a = d.decide({"text": text}, {"label": {"type": "choice", "instructions": Q_CHOICE, "criteria": crit}}, lang="en")["label"]
+        return [float(a["probabilities"][k]) for k in crit]
+    def binary(text, labels):                      # one prompt per question (canonical path)
+        qs = {"q%03d" % i: {"type": "noul", "instructions": Q_BINARY.format(label=l)} for i, l in enumerate(labels)}
+        a = d.decide({"text": text}, qs, lang="en")
+        return [float(a["q%03d" % i]["noul"]) for i in range(len(labels))]
+elif BACKEND == "julia":
+    from julia import load_model
+    from julia.router import Router
+    eng = load_model(os.environ["JULIA"], device="cuda", strict_encoding=True, max_length=8192, head_length=512)
+    router = Router(eng, width=20, survivors=2)    # the README's documented setting for > 20 options
+    def choice(text, labels):
+        crit = {"label_%03d" % i: l for i, l in enumerate(labels)}
+        a = eng.predict(state={"text": text}, questions={"label": {"type": "choice", "instructions": Q_CHOICE, "criteria": crit}})["answers"]["label"]
+        return [float(a["probabilities"][k]) for k in crit]
+    def routed(text, labels):
+        r = router.route({"state": {"text": text}, "question": Q_CHOICE, "options": list(labels), "type": "choice"})
+        return r.index, {"candidates": list(r.candidates), "candidate_probabilities": list(r.probabilities), "rounds": r.rounds,
+                         "model_rows": r.model_rows, "probability_scope": r.probability_scope}
+    def binary(text, labels):                      # independent questions batched in one call
+        qs = {"q%03d" % i: {"type": "noul", "instructions": Q_BINARY.format(label=l)} for i, l in enumerate(labels)}
+        a = eng.predict(state={"text": text}, questions=qs)["answers"]
+        return [float(a["q%03d" % i]["noul"]) for i in range(len(labels))]
+elif BACKEND == "laya":
+    import laya
+    ag = laya.load(os.environ["LAYA"])
+    def choice(text, labels):                      # criteria as a list: options are the label texts themselves
+        a = ag.predict({"text": text}, {"label": {"type": "choice", "instructions": Q_CHOICE, "criteria": list(labels)}}, max_len=8192)["answers"]["label"]
+        return [float(a["probabilities"][l]) for l in labels]
+    def binary(text, labels):
+        qs = {"q%03d" % i: {"type": "noul", "instructions": Q_BINARY.format(label=l)} for i, l in enumerate(labels)}
+        a = ag.predict({"text": text}, qs, max_len=8192)["answers"]
+        return [float(a["q%03d" % i]["noul"]) for i in range(len(labels))]
+elif BACKEND == "gliner":
+    from gliner2.classification import ClassificationSchema, Classifier
+    clf = Classifier.from_pretrained("fastino/gliner2.5-multi-v1", revision="235cf92d6d4318da9bfca0d08975c8fa7250d13b", device="cpu").eval()
+    if os.environ.get("GLINER_DEVICE") == "cuda":
+        clf = clf.to("cuda")
+    def choice(text, labels):
+        r = clf.classify(text, ClassificationSchema().single("label", list(labels)))
+        return [float(r.probabilities("label")[l]) for l in labels]
+    def binary(text, labels):                      # task name = the binary question, labels yes / no
+        out = []
+        for l in labels:
+            q = Q_BINARY.format(label=l)
+            out.append(float(clf.classify(text, ClassificationSchema().single(q, ["yes", "no"])).probabilities(q)["yes"]))
+        return out
+
+LIMIT = {"ines": 26, "julia": 20, "laya": None, "gliner": None}[BACKEND]
+
+def native(text, labels):
+    if LIMIT is None or len(labels) <= LIMIT:
+        p = choice(text, labels)
+        return max(range(len(p)), key=p.__getitem__), {"probabilities": p, "strategy": "single request", "calls": 1}
+    if BACKEND == "julia":
+        i, meta = routed(text, labels)
+        return i, dict(meta, strategy="julia Router(width=20, survivors=2)")
+    raise SystemExit("no native strategy for %s with %d labels (Ines-1 > 26 uses the knockout ablation)" % (BACKEND, len(labels)))
+
+def ovr(text, labels):
+    p = binary(text, labels)
+    s = [math.log(min(max(x, EPS), 1 - EPS)) - math.log(1 - min(max(x, EPS), 1 - EPS)) for x in p]
+    best = max(s); ties = sum(1 for v in s if v == best)
+    pred = s.index(best)                            # ties -> lowest label index (counted)
+    z = [math.exp(v - best) for v in s]; tot = sum(z)
+    return pred, {"p_yes": p, "log_odds": s, "derived_probabilities": [v / tot for v in z], "ties_at_max": ties, "calls": len(labels)}
+
+done = set()
+if os.path.exists(OUT):
+    done = {json.loads(l)["example_id"] for l in open(OUT)}
+ex = [json.loads(l) for l in open(MAN)]
+with open(OUT, "a") as f:
+    for e in ex:
+        if e["example_id"] in done:
+            continue
+        L = e["labels"]
+        if TRACK == "native" and BACKEND == "ines" and len(L) > 26:
+            continue                                # Ines-1 has no native > 26 interface: see ablation (knockout)
+        t = time.perf_counter()
+        try:
+            pred, meta = (native if TRACK == "native" else ovr)(e["text"], L); err = None
+        except SystemExit:
+            raise
+        except Exception as x:                      # failures stay in the denominator
+            pred, meta, err = -1, {}, repr(x)[:300]
+        f.write(json.dumps({"backend": BACKEND, "track": TRACK, "dataset": e["dataset"], "example_id": e["example_id"],
+                            "target_index": e["target_index"], "predicted_index": pred, "n_labels": len(L),
+                            "latency_seconds": time.perf_counter() - t, "error": err, **meta}) + "\n")
+        f.flush()
+print("done", BACKEND, TRACK)

eval/btzsc22/code/run_tracks.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+# btzsc22-ext-v1: Track A (model-specific classification) and Track B (one-vs-rest) with the frozen run_track.py.
+# Set by the caller (paths generalised for publication; the executed script differed only in these locations):
+#   INES, JULIA, LAYA          local snapshots of the three decision models (GLiNER2.5 is fetched by gliner2)
+#   PY_INES PY_JULIA PY_LAYA PY_GLINER   python of each isolated environment (see ../PROTOCOL.md)
+#   INES_PYTHONPATH            extra path giving Ines-1's environment the `safetensors` package, if needed
+M=runs/btzsc22/manifest.jsonl; O=runs/btzsc22/v1; mkdir -p $O
+export USE_TF=0
+run() { local b=$1; shift; "$@" run_track.py $b $M $O/$b.native.jsonl native && "$@" run_track.py $b $M $O/$b.ovr.jsonl ovr; echo "$b-DONE"; }
+( PYTHONPATH=$INES_PYTHONPATH run ines $PY_INES ) > $O/ines.log 2>&1 &
+( TORCHDYNAMO_DISABLE=1 TORCH_DISABLE_NATIVE_JIT=1 run julia $PY_JULIA ) > $O/julia.log 2>&1 &
+( TORCHDYNAMO_DISABLE=1 TORCH_DISABLE_NATIVE_JIT=1 run laya $PY_LAYA ) > $O/laya.log 2>&1 &
+( GLINER_DEVICE=cuda run gliner $PY_GLINER ) > $O/gliner.log 2>&1 &
+wait; echo ALL-DONE

eval/btzsc22/code/score_final.py

@@ -0,0 +1,121 @@
+"""btzsc22-ext-v1 analysis (frozen before reading Track B). argv: run_dir(v1) exploratory_dir out_dir
+Run with the gliner venv (uses jev_benchmarks.metrics unchanged)."""
+import csv, json, sys
+from pathlib import Path
+import numpy as np
+from jev_benchmarks.metrics import score_predictions, _macro_f1
+from jev_benchmarks.models import Prediction
+
+V1, EXP, OUT = Path(sys.argv[1]), Path(sys.argv[2]), Path(sys.argv[3]); OUT.mkdir(parents=True, exist_ok=True)
+MODELS = ["ines", "julia", "laya", "gliner"]
+NAME = {"ines": "Ines-1", "julia": "Julia-1", "laya": "laya-multilingual", "gliner": "GLiNER2.5"}
+IN_MIX = {"amazonpolarity", "banking77", "massive", "wikitoxic_insult", "wikitoxic_obscene", "wikitoxic_threat", "wikitoxic_toxicaggregated"}
+man = [json.loads(l) for l in open(EXP / "manifest.jsonl")]
+DS = list(dict.fromkeys(e["dataset"] for e in man)); K = {e["dataset"]: len(e["labels"]) for e in man}
+IDS = {d: [e["example_id"] for e in man if e["dataset"] == d] for d in DS}
+T = {d: np.array([e["target_index"] for e in man if e["dataset"] == d]) for d in DS}
+LAB = {e["example_id"]: tuple(e["labels"]) for e in man}
+rd = lambda p: {json.loads(l)["example_id"]: json.loads(l) for l in open(p)} if Path(p).exists() else {}
+native = {m: rd(V1 / ("%s.native.jsonl" % m)) for m in MODELS}
+ovr = {m: rd(V1 / ("%s.ovr.jsonl" % m)) for m in MODELS}
+ines_k20, ines_k2 = rd(EXP / "ines.k20.jsonl"), rd(EXP / "ines.k2.jsonl")
+
+def preds(src, d):
+    return np.array([(src[i]["predicted_index"] if i in src and not src[i].get("error") else -1) for i in IDS[d]])
+def trackA(m, d):
+    if m == "ines" and K[d] > 26:
+        return preds(ines_k20, d), "knockout-20 (no native > 26 interface)"
+    r = native[m]; s = "single request"
+    if m == "julia" and K[d] > 20: s = "Router(width=20, survivors=2)"
+    return preds(r, d), s
+P = {"A": {m: {d: trackA(m, d)[0] for d in DS} for m in MODELS}, "B": {m: {d: preds(ovr[m], d) for d in DS} for m in MODELS}}
+missing = {tr: {m: int(sum((P[tr][m][d] < 0).sum() for d in DS)) for m in MODELS} for tr in P}
+f1 = lambda tr, m, d, idx=None: _macro_f1(T[d] if idx is None else T[d][idx], P[tr][m][d] if idx is None else P[tr][m][d][idx], K[d])
+acc = lambda tr, m, d: float(np.mean(P[tr][m][d] == T[d]))
+
+GROUPS = {"all 22": DS, "2 labels (13)": [d for d in DS if K[d] == 2], "3-20 labels (4)": [d for d in DS if 2 < K[d] <= 20],
+          "21-72 labels (5)": [d for d in DS if K[d] > 20], "absent from Ines-1 decision FT mixture (15)": [d for d in DS if d not in IN_MIX],
+          "families in Ines-1 decision FT mixture (7)": [d for d in DS if d in IN_MIX]}
+rng = np.random.default_rng(20260917)
+STRATA = {d: [np.where(T[d] == c)[0] for c in np.unique(T[d])] for d in DS}
+def hboot(sel, pairs, B=2000):
+    """paired hierarchical bootstrap: resample datasets, then examples stratified by target; equal-weight mean macro-F1"""
+    out = {p: [] for p in pairs}
+    for _ in range(B):
+        draw = rng.integers(0, len(sel), len(sel))
+        acc_ = {p: [] for p in pairs}
+        for j in draw:
+            d = sel[j]; idx = np.concatenate([rng.choice(c, len(c)) for c in STRATA[d]])
+            cache = {}
+            for p in pairs:
+                (ta, ma), (tb, mb) = p
+                for key in ((ta, ma), (tb, mb)):
+                    if key not in cache: cache[key] = f1(key[0], key[1], d, idx)
+                acc_[p].append(cache[(tb, mb)] - cache[(ta, ma)])
+        for p in pairs: out[p].append(np.mean(acc_[p]))
+    res = {}
+    for p, v in out.items():
+        (ta, ma), (tb, mb) = p
+        obs = float(np.mean([f1(tb, mb, d) - f1(ta, ma, d) for d in sel]))
+        res["%s:%s - %s:%s" % (tb, mb, ta, ma)] = {"difference": obs, "ci95": [float(np.quantile(v, .025)), float(np.quantile(v, .975))]}
+    return res
+
+R = {"protocol": "btzsc22-ext-v1", "missing_predictions": missing, "per_dataset": {}, "global": {}, "bootstrap": {}, "calibration": {},
+     "ines_ablation": {}, "anchors": {}, "ovr_ties": {}}
+for d in DS:
+    R["per_dataset"][d] = {"labels": K[d], "in_ines_ft_mixture": d in IN_MIX,
+                           **{tr: {m: {"accuracy": acc(tr, m, d), "macro_f1": f1(tr, m, d)} for m in MODELS} for tr in ("A", "B")},
+                           "trackA_strategy": {m: trackA(m, d)[1] for m in MODELS}}
+for g, sel in GROUPS.items():
+    R["global"][g] = {tr: {m: float(np.mean([f1(tr, m, d) for d in sel])) for m in MODELS} for tr in ("A", "B")}
+cross = [(("A", "ines"), ("A", m)) for m in MODELS[1:]] + [(("B", "ines"), ("B", m)) for m in MODELS[1:]] + [(("A", m), ("B", m)) for m in MODELS]
+for g, sel in GROUPS.items():
+    R["bootstrap"][g] = hboot(sel, cross, B=2000 if g == "all 22" else 1000)
+
+def calib(srcs, field, sel):
+    out = {}
+    for m in MODELS:
+        per = []
+        for d in sel:
+            rows = []
+            for i in IDS[d]:
+                r = srcs[m].get(i)
+                if not r or r.get("error") or not r.get(field): break
+                rows.append(Prediction(experiment_id="x", backend=m, model_requested=m, model_resolved=m, dataset=d, example_id=i,
+                                       target_index=int(T[d][IDS[d].index(i)]), predicted_index=r["predicted_index"], labels=LAB[i],
+                                       probabilities=tuple(r[field]), latency_seconds=0.0))
+            else:
+                per.append(score_predictions(rows))
+        if per:
+            out[m] = {k: float(np.mean([s[k] for s in per])) for k in ("brier", "nll", "ece", "coverage_at_error_budget", "mean_confidence")} | {"datasets": len(per)}
+    return out
+native_full = [d for d in DS if K[d] <= 20]
+R["calibration"]["native (Track A, full native distributions, <=20 labels, 17 datasets)"] = calib(native, "probabilities", native_full)
+R["calibration"]["derived (Track B softmax of log-odds, same 17 datasets)"] = calib(ovr, "derived_probabilities", native_full)
+R["calibration"]["derived (Track B softmax of log-odds, all 22 datasets)"] = calib(ovr, "derived_probabilities", DS)
+R["ovr_ties"] = {m: {"examples_with_tie_at_max": sum(1 for r in ovr[m].values() if r.get("ties_at_max", 1) > 1), "n": len(ovr[m])} for m in MODELS}
+
+multi = [d for d in DS if K[d] > 2]
+for d in multi:
+    row = {"native(<=26)": acc("A", "ines", d) if K[d] <= 26 else None, "knockout-20": float(np.mean(preds(ines_k20, d) == T[d])) if K[d] > 20 else None,
+           "knockout-2": float(np.mean(preds(ines_k2, d) == T[d])), "one-vs-rest": acc("B", "ines", d)}
+    rowf = {"native(<=26)": _macro_f1(T[d], preds(native["ines"], d), K[d]) if K[d] <= 26 else None,
+            "knockout-20": _macro_f1(T[d], preds(ines_k20, d), K[d]) if K[d] > 20 else None,
+            "knockout-2": _macro_f1(T[d], preds(ines_k2, d), K[d]), "one-vs-rest": f1("B", "ines", d)}
+    R["ines_ablation"][d] = {"labels": K[d], "accuracy": row, "macro_f1": rowf}
+
+pub = {"agnews": 0.70, "emotiondair": 0.44, "banking77": 0.61}
+R["anchors"]["GLiNER2.5 Track A vs published pilot (accuracy)"] = {d: {"ours": acc("A", "gliner", d), "published": v} for d, v in pub.items()}
+card = {"agnews": 0.94, "emotiondair": 0.86, "banking77": 0.64}
+R["anchors"]["Julia-1 Track A vs its model card (accuracy)"] = {d: {"ours": acc("A", "julia", d), "card": v, "ours_strategy": trackA("julia", d)[1]} for d, v in card.items()}
+
+json.dump(R, open(OUT / "results.json", "w"), indent=1)
+with open(OUT / "per_dataset.csv", "w", newline="") as f:
+    w = csv.writer(f); w.writerow(["dataset", "labels", "in_ines1_decision_ft_mixture"] + ["%s_%s_%s" % (tr, m, k) for tr in ("A", "B") for m in MODELS for k in ("accuracy", "macro_f1")] + ["A_strategy_ines", "A_strategy_julia"])
+    for d in DS:
+        x = R["per_dataset"][d]
+        w.writerow([d, K[d], d in IN_MIX] + ["%.4f" % x[tr][m][k] for tr in ("A", "B") for m in MODELS for k in ("accuracy", "macro_f1")] + [x["trackA_strategy"]["ines"], x["trackA_strategy"]["julia"]])
+print("missing", missing)
+for g in GROUPS:
+    print("%-45s A %s | B %s" % (g, "  ".join("%s %.3f" % (m, R["global"][g]["A"][m]) for m in MODELS), "  ".join("%s %.3f" % (m, R["global"][g]["B"][m]) for m in MODELS)))
+for k, v in R["bootstrap"]["all 22"].items(): print("  %-28s %+.3f [%+.3f, %+.3f]" % (k, v["difference"], *v["ci95"]))

eval/btzsc22/per_dataset.csv

@@ -0,0 +1,23 @@
+dataset,labels,in_ines1_decision_ft_mixture,A_ines_accuracy,A_ines_macro_f1,A_julia_accuracy,A_julia_macro_f1,A_laya_accuracy,A_laya_macro_f1,A_gliner_accuracy,A_gliner_macro_f1,B_ines_accuracy,B_ines_macro_f1,B_julia_accuracy,B_julia_macro_f1,B_laya_accuracy,B_laya_macro_f1,B_gliner_accuracy,B_gliner_macro_f1,A_strategy_ines,A_strategy_julia
+agnews,4,False,0.5900,0.5843,0.9400,0.9399,0.9400,0.9393,0.7000,0.6589,0.4200,0.3265,0.4200,0.3995,0.8900,0.8890,0.4500,0.3986,single request,single request
+emotiondair,6,False,0.4300,0.3904,0.8600,0.8557,0.3700,0.3527,0.4400,0.4070,0.3400,0.2948,0.2600,0.2543,0.4300,0.4279,0.3500,0.3193,single request,single request
+banking77,72,True,0.3900,0.3427,0.6300,0.5704,0.0200,0.0006,0.6100,0.5693,0.2900,0.2235,0.0900,0.0472,0.2200,0.1880,0.4100,0.3788,knockout-20 (no native > 26 interface),"Router(width=20, survivors=2)"
+amazonpolarity,2,True,0.7800,0.7756,0.6200,0.6162,0.8400,0.8368,0.8800,0.8800,0.6500,0.6179,0.5800,0.5716,0.9100,0.9098,0.8300,0.8249,single request,single request
+imdb,2,False,0.7500,0.7500,0.6800,0.6799,0.7300,0.7149,0.8000,0.7947,0.6900,0.6808,0.5400,0.5354,0.8100,0.8091,0.7300,0.7175,single request,single request
+appreviews,2,False,0.7700,0.7660,0.6500,0.6396,0.7800,0.7756,0.8900,0.8895,0.8100,0.8095,0.5100,0.4954,0.8900,0.8897,0.8000,0.7960,single request,single request
+yelpreviews,2,False,0.7200,0.6997,0.7300,0.7278,0.9000,0.8996,0.9000,0.8994,0.6800,0.6435,0.5100,0.5088,0.9300,0.9298,0.7400,0.7211,single request,single request
+rottentomatoes,2,False,0.5900,0.5671,0.6600,0.6264,0.6900,0.6808,0.7100,0.7064,0.6400,0.6347,0.4600,0.4325,0.7400,0.7400,0.6700,0.6548,single request,single request
+financialphrasebank,3,False,0.5100,0.4891,0.4500,0.3711,0.8100,0.8085,0.6900,0.6821,0.4500,0.3886,0.2500,0.1744,0.9100,0.9097,0.5500,0.4785,single request,single request
+empathetic,32,False,0.2500,0.1909,0.1700,0.1594,0.0600,0.0255,0.3300,0.2807,0.1800,0.1442,0.0500,0.0390,0.3300,0.2873,0.1400,0.1304,knockout-20 (no native > 26 interface),"Router(width=20, survivors=2)"
+biasframes_intent,2,False,0.6000,0.5998,0.5700,0.5665,0.6100,0.5920,0.5800,0.5739,0.5900,0.5524,0.5900,0.5806,0.6600,0.6550,0.5200,0.3912,single request,single request
+massive,59,True,0.3500,0.2502,0.2700,0.1802,0.0100,0.0010,0.4400,0.3827,0.2000,0.1528,0.0200,0.0122,0.3500,0.3255,0.3000,0.2354,knockout-20 (no native > 26 interface),"Router(width=20, survivors=2)"
+yahootopics,10,False,0.5000,0.4915,0.3900,0.3675,0.1500,0.1369,0.4100,0.4170,0.1800,0.1231,0.1200,0.1127,0.3700,0.3650,0.3200,0.3175,single request,single request
+trueteacher,2,False,0.5600,0.5376,0.5200,0.5152,0.5600,0.4658,0.5300,0.5242,0.5000,0.3333,0.5000,0.4346,0.4800,0.4207,0.5400,0.4524,single request,single request
+manifesto,56,False,0.1100,0.0750,0.1000,0.0789,0.0300,0.0077,0.0200,0.0007,0.0500,0.0329,0.0200,0.0149,0.1100,0.0903,0.0600,0.0136,knockout-20 (no native > 26 interface),"Router(width=20, survivors=2)"
+capsotu,21,False,0.3600,0.3121,0.4500,0.4452,0.0500,0.0050,0.3000,0.2524,0.2100,0.1784,0.1000,0.0799,0.2700,0.2800,0.1700,0.1837,single request,"Router(width=20, survivors=2)"
+biasframes_offensive,2,False,0.5200,0.4956,0.5400,0.5119,0.6200,0.6194,0.4600,0.4375,0.4800,0.4800,0.4500,0.4068,0.6900,0.6885,0.5200,0.3763,single request,single request
+biasframes_sex,2,False,0.3400,0.3389,0.5600,0.5556,0.7200,0.7083,0.5300,0.3967,0.5300,0.5192,0.5600,0.5098,0.7900,0.7900,0.5600,0.4762,single request,single request
+wikitoxic_insult,2,True,0.7700,0.7694,0.5600,0.5593,0.8500,0.8482,0.7000,0.6900,0.6900,0.6900,0.5400,0.5308,0.8000,0.7960,0.5800,0.5000,single request,single request
+wikitoxic_obscene,2,True,0.4400,0.3455,0.5600,0.5572,0.6500,0.6419,0.5200,0.3912,0.7300,0.7300,0.6300,0.6190,0.8300,0.8279,0.6200,0.5703,single request,single request
+wikitoxic_threat,2,True,0.4700,0.3498,0.4700,0.4674,0.9100,0.9100,0.5300,0.3967,0.7300,0.7254,0.5500,0.5423,0.7700,0.7614,0.5200,0.3763,single request,single request
+wikitoxic_toxicaggregated,2,True,0.3300,0.3049,0.5700,0.5502,0.7000,0.6921,0.6200,0.5766,0.7700,0.7681,0.5700,0.5572,0.8700,0.8678,0.6200,0.5766,single request,single request

eval/btzsc22/results.json

@@ -0,0 +1,1843 @@
+{
+ "protocol": "btzsc22-ext-v1",
+ "missing_predictions": {
+  "A": {
+   "ines": 0,
+   "julia": 0,
+   "laya": 0,
+   "gliner": 0
+  },
+  "B": {
+   "ines": 0,
+   "julia": 0,
+   "laya": 0,
+   "gliner": 0
+  }
+ },
+ "per_dataset": {
+  "agnews": {
+   "labels": 4,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.59,
+     "macro_f1": 0.5843451916892106
+    },
+    "julia": {
+     "accuracy": 0.94,
+     "macro_f1": 0.9399173646545361
+    },
+    "laya": {
+     "accuracy": 0.94,
+     "macro_f1": 0.939271525304066
+    },
+    "gliner": {
+     "accuracy": 0.7,
+     "macro_f1": 0.6588522320524076
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.42,
+     "macro_f1": 0.32652938961350175
+    },
+    "julia": {
+     "accuracy": 0.42,
+     "macro_f1": 0.399474348626891
+    },
+    "laya": {
+     "accuracy": 0.89,
+     "macro_f1": 0.8889659795261577
+    },
+    "gliner": {
+     "accuracy": 0.45,
+     "macro_f1": 0.39861358684888093
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "emotiondair": {
+   "labels": 6,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.43,
+     "macro_f1": 0.39042230395613853
+    },
+    "julia": {
+     "accuracy": 0.86,
+     "macro_f1": 0.8556934538891329
+    },
+    "laya": {
+     "accuracy": 0.37,
+     "macro_f1": 0.35268000920174836
+    },
+    "gliner": {
+     "accuracy": 0.44,
+     "macro_f1": 0.4070167592505414
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.34,
+     "macro_f1": 0.29478472669962036
+    },
+    "julia": {
+     "accuracy": 0.26,
+     "macro_f1": 0.25425384800384804
+    },
+    "laya": {
+     "accuracy": 0.43,
+     "macro_f1": 0.4279340742423987
+    },
+    "gliner": {
+     "accuracy": 0.35,
+     "macro_f1": 0.3193197964297709
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "banking77": {
+   "labels": 72,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.39,
+     "macro_f1": 0.34265873015873016
+    },
+    "julia": {
+     "accuracy": 0.63,
+     "macro_f1": 0.5703703703703703
+    },
+    "laya": {
+     "accuracy": 0.02,
+     "macro_f1": 0.0005611672278338945
+    },
+    "gliner": {
+     "accuracy": 0.61,
+     "macro_f1": 0.569268077601411
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.29,
+     "macro_f1": 0.2234898589065256
+    },
+    "julia": {
+     "accuracy": 0.09,
+     "macro_f1": 0.04716343327454439
+    },
+    "laya": {
+     "accuracy": 0.22,
+     "macro_f1": 0.18796296296296294
+    },
+    "gliner": {
+     "accuracy": 0.41,
+     "macro_f1": 0.3787818662818662
+    }
+   },
+   "trackA_strategy": {
+    "ines": "knockout-20 (no native > 26 interface)",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "amazonpolarity": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.78,
+     "macro_f1": 0.7756017951856384
+    },
+    "julia": {
+     "accuracy": 0.62,
+     "macro_f1": 0.6161616161616161
+    },
+    "laya": {
+     "accuracy": 0.84,
+     "macro_f1": 0.8368013055895553
+    },
+    "gliner": {
+     "accuracy": 0.88,
+     "macro_f1": 0.8799519807923168
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.65,
+     "macro_f1": 0.6178622120318812
+    },
+    "julia": {
+     "accuracy": 0.58,
+     "macro_f1": 0.5716034271725826
+    },
+    "laya": {
+     "accuracy": 0.91,
+     "macro_f1": 0.9097744360902256
+    },
+    "gliner": {
+     "accuracy": 0.83,
+     "macro_f1": 0.8249407887962105
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "imdb": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.75,
+     "macro_f1": 0.74997499749975
+    },
+    "julia": {
+     "accuracy": 0.68,
+     "macro_f1": 0.6798719487795117
+    },
+    "laya": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7149192271143491
+    },
+    "gliner": {
+     "accuracy": 0.8,
+     "macro_f1": 0.7947454844006567
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6807743795695603
+    },
+    "julia": {
+     "accuracy": 0.54,
+     "macro_f1": 0.5353535353535354
+    },
+    "laya": {
+     "accuracy": 0.81,
+     "macro_f1": 0.8090644156366193
+    },
+    "gliner": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7175436761167486
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "appreviews": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.77,
+     "macro_f1": 0.7660461804495982
+    },
+    "julia": {
+     "accuracy": 0.65,
+     "macro_f1": 0.6395839769333744
+    },
+    "laya": {
+     "accuracy": 0.78,
+     "macro_f1": 0.7756017951856384
+    },
+    "gliner": {
+     "accuracy": 0.89,
+     "macro_f1": 0.8894583458948849
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.81,
+     "macro_f1": 0.8095238095238095
+    },
+    "julia": {
+     "accuracy": 0.51,
+     "macro_f1": 0.49541756770672435
+    },
+    "laya": {
+     "accuracy": 0.89,
+     "macro_f1": 0.8897243107769424
+    },
+    "gliner": {
+     "accuracy": 0.8,
+     "macro_f1": 0.7960016319869441
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "yelpreviews": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.72,
+     "macro_f1": 0.6996996996996997
+    },
+    "julia": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7277951406391774
+    },
+    "laya": {
+     "accuracy": 0.9,
+     "macro_f1": 0.8996386993175431
+    },
+    "gliner": {
+     "accuracy": 0.9,
+     "macro_f1": 0.8993558776167472
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.68,
+     "macro_f1": 0.64349376114082
+    },
+    "julia": {
+     "accuracy": 0.51,
+     "macro_f1": 0.5087719298245614
+    },
+    "laya": {
+     "accuracy": 0.93,
+     "macro_f1": 0.9298245614035088
+    },
+    "gliner": {
+     "accuracy": 0.74,
+     "macro_f1": 0.7211497211497211
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "rottentomatoes": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.59,
+     "macro_f1": 0.5670995670995671
+    },
+    "julia": {
+     "accuracy": 0.66,
+     "macro_f1": 0.6263736263736264
+    },
+    "laya": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6807743795695603
+    },
+    "gliner": {
+     "accuracy": 0.71,
+     "macro_f1": 0.7064480210547626
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.64,
+     "macro_f1": 0.6347402597402598
+    },
+    "julia": {
+     "accuracy": 0.46,
+     "macro_f1": 0.43253467843631777
+    },
+    "laya": {
+     "accuracy": 0.74,
+     "macro_f1": 0.74
+    },
+    "gliner": {
+     "accuracy": 0.67,
+     "macro_f1": 0.6547756041426928
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "financialphrasebank": {
+   "labels": 3,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.51,
+     "macro_f1": 0.48911853494169083
+    },
+    "julia": {
+     "accuracy": 0.45,
+     "macro_f1": 0.37110209601081806
+    },
+    "laya": {
+     "accuracy": 0.81,
+     "macro_f1": 0.8084906388321863
+    },
+    "gliner": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6821411017203141
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.45,
+     "macro_f1": 0.388558378810103
+    },
+    "julia": {
+     "accuracy": 0.25,
+     "macro_f1": 0.17441737846672992
+    },
+    "laya": {
+     "accuracy": 0.91,
+     "macro_f1": 0.9096611274030629
+    },
+    "gliner": {
+     "accuracy": 0.55,
+     "macro_f1": 0.47845237675746155
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "empathetic": {
+   "labels": 32,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.25,
+     "macro_f1": 0.1909003551273288
+    },
+    "julia": {
+     "accuracy": 0.17,
+     "macro_f1": 0.15942460317460316
+    },
+    "laya": {
+     "accuracy": 0.06,
+     "macro_f1": 0.02546768707482993
+    },
+    "gliner": {
+     "accuracy": 0.33,
+     "macro_f1": 0.2807043650793651
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.18,
+     "macro_f1": 0.14416745535166586
+    },
+    "julia": {
+     "accuracy": 0.05,
+     "macro_f1": 0.039042207792207795
+    },
+    "laya": {
+     "accuracy": 0.33,
+     "macro_f1": 0.2872519841269841
+    },
+    "gliner": {
+     "accuracy": 0.14,
+     "macro_f1": 0.13043154761904763
+    }
+   },
+   "trackA_strategy": {
+    "ines": "knockout-20 (no native > 26 interface)",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "biasframes_intent": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.6,
+     "macro_f1": 0.5998399359743898
+    },
+    "julia": {
+     "accuracy": 0.57,
+     "macro_f1": 0.5664885573142454
+    },
+    "laya": {
+     "accuracy": 0.61,
+     "macro_f1": 0.5920075321686369
+    },
+    "gliner": {
+     "accuracy": 0.58,
+     "macro_f1": 0.5738636363636364
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.59,
+     "macro_f1": 0.5523528769516323
+    },
+    "julia": {
+     "accuracy": 0.59,
+     "macro_f1": 0.5805626598465473
+    },
+    "laya": {
+     "accuracy": 0.66,
+     "macro_f1": 0.6550324675324675
+    },
+    "gliner": {
+     "accuracy": 0.52,
+     "macro_f1": 0.39117199391172
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "massive": {
+   "labels": 59,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.35,
+     "macro_f1": 0.2502433536331841
+    },
+    "julia": {
+     "accuracy": 0.27,
+     "macro_f1": 0.1801721818670971
+    },
+    "laya": {
+     "accuracy": 0.01,
+     "macro_f1": 0.0009685230024213075
+    },
+    "gliner": {
+     "accuracy": 0.44,
+     "macro_f1": 0.38272522334474407
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.2,
+     "macro_f1": 0.15282852740479855
+    },
+    "julia": {
+     "accuracy": 0.02,
+     "macro_f1": 0.01224105461393597
+    },
+    "laya": {
+     "accuracy": 0.35,
+     "macro_f1": 0.3254640839386602
+    },
+    "gliner": {
+     "accuracy": 0.3,
+     "macro_f1": 0.2353510895883777
+    }
+   },
+   "trackA_strategy": {
+    "ines": "knockout-20 (no native > 26 interface)",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "yahootopics": {
+   "labels": 10,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.5,
+     "macro_f1": 0.4915120854091442
+    },
+    "julia": {
+     "accuracy": 0.39,
+     "macro_f1": 0.3675051008447593
+    },
+    "laya": {
+     "accuracy": 0.15,
+     "macro_f1": 0.13690074545337702
+    },
+    "gliner": {
+     "accuracy": 0.41,
+     "macro_f1": 0.4169864449276214
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.18,
+     "macro_f1": 0.12307189542483658
+    },
+    "julia": {
+     "accuracy": 0.12,
+     "macro_f1": 0.11272908345326083
+    },
+    "laya": {
+     "accuracy": 0.37,
+     "macro_f1": 0.36499120762278653
+    },
+    "gliner": {
+     "accuracy": 0.32,
+     "macro_f1": 0.31750090466808734
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "trueteacher": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.56,
+     "macro_f1": 0.537620849096259
+    },
+    "julia": {
+     "accuracy": 0.52,
+     "macro_f1": 0.5151515151515151
+    },
+    "laya": {
+     "accuracy": 0.56,
+     "macro_f1": 0.46576007770762506
+    },
+    "gliner": {
+     "accuracy": 0.53,
+     "macro_f1": 0.5242433444680635
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.5,
+     "macro_f1": 0.3333333333333333
+    },
+    "julia": {
+     "accuracy": 0.5,
+     "macro_f1": 0.43464495703301675
+    },
+    "laya": {
+     "accuracy": 0.48,
+     "macro_f1": 0.4206773618538324
+    },
+    "gliner": {
+     "accuracy": 0.54,
+     "macro_f1": 0.45238095238095233
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "manifesto": {
+   "labels": 56,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.11,
+     "macro_f1": 0.07504578754578754
+    },
+    "julia": {
+     "accuracy": 0.1,
+     "macro_f1": 0.0788690476190476
+    },
+    "laya": {
+     "accuracy": 0.03,
+     "macro_f1": 0.007711038961038961
+    },
+    "gliner": {
+     "accuracy": 0.02,
+     "macro_f1": 0.0007072135785007072
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.05,
+     "macro_f1": 0.032902735562310034
+    },
+    "julia": {
+     "accuracy": 0.02,
+     "macro_f1": 0.01488095238095238
+    },
+    "laya": {
+     "accuracy": 0.11,
+     "macro_f1": 0.09031507339778015
+    },
+    "gliner": {
+     "accuracy": 0.06,
+     "macro_f1": 0.013575543120473996
+    }
+   },
+   "trackA_strategy": {
+    "ines": "knockout-20 (no native > 26 interface)",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "capsotu": {
+   "labels": 21,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.36,
+     "macro_f1": 0.3120540695728665
+    },
+    "julia": {
+     "accuracy": 0.45,
+     "macro_f1": 0.4451588094445238
+    },
+    "laya": {
+     "accuracy": 0.05,
+     "macro_f1": 0.005012531328320802
+    },
+    "gliner": {
+     "accuracy": 0.3,
+     "macro_f1": 0.2524050024050024
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.21,
+     "macro_f1": 0.17844985702128557
+    },
+    "julia": {
+     "accuracy": 0.1,
+     "macro_f1": 0.0798883656026513
+    },
+    "laya": {
+     "accuracy": 0.27,
+     "macro_f1": 0.2799847801788137
+    },
+    "gliner": {
+     "accuracy": 0.17,
+     "macro_f1": 0.18369192344347623
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "Router(width=20, survivors=2)",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "biasframes_offensive": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.52,
+     "macro_f1": 0.49558638083228246
+    },
+    "julia": {
+     "accuracy": 0.54,
+     "macro_f1": 0.5118845500848896
+    },
+    "laya": {
+     "accuracy": 0.62,
+     "macro_f1": 0.6193910256410255
+    },
+    "gliner": {
+     "accuracy": 0.46,
+     "macro_f1": 0.4375
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.48,
+     "macro_f1": 0.48
+    },
+    "julia": {
+     "accuracy": 0.45,
+     "macro_f1": 0.40675223816201056
+    },
+    "laya": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6884735202492211
+    },
+    "gliner": {
+     "accuracy": 0.52,
+     "macro_f1": 0.37629937629937626
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "biasframes_sex": {
+   "labels": 2,
+   "in_ines_ft_mixture": false,
+   "A": {
+    "ines": {
+     "accuracy": 0.34,
+     "macro_f1": 0.3389423076923077
+    },
+    "julia": {
+     "accuracy": 0.56,
+     "macro_f1": 0.5555555555555556
+    },
+    "laya": {
+     "accuracy": 0.72,
+     "macro_f1": 0.7083333333333334
+    },
+    "gliner": {
+     "accuracy": 0.53,
+     "macro_f1": 0.39673982800667434
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.53,
+     "macro_f1": 0.5191815856777494
+    },
+    "julia": {
+     "accuracy": 0.56,
+     "macro_f1": 0.5098039215686274
+    },
+    "laya": {
+     "accuracy": 0.79,
+     "macro_f1": 0.78997899789979
+    },
+    "gliner": {
+     "accuracy": 0.56,
+     "macro_f1": 0.47619047619047616
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "wikitoxic_insult": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.77,
+     "macro_f1": 0.7694235588972431
+    },
+    "julia": {
+     "accuracy": 0.56,
+     "macro_f1": 0.5592948717948718
+    },
+    "laya": {
+     "accuracy": 0.85,
+     "macro_f1": 0.8481627695110842
+    },
+    "gliner": {
+     "accuracy": 0.7,
+     "macro_f1": 0.6899545266639107
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.69,
+     "macro_f1": 0.6899689968996899
+    },
+    "julia": {
+     "accuracy": 0.54,
+     "macro_f1": 0.5308037535699714
+    },
+    "laya": {
+     "accuracy": 0.8,
+     "macro_f1": 0.7960016319869441
+    },
+    "gliner": {
+     "accuracy": 0.58,
+     "macro_f1": 0.5
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "wikitoxic_obscene": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.44,
+     "macro_f1": 0.34548854604955587
+    },
+    "julia": {
+     "accuracy": 0.56,
+     "macro_f1": 0.5571658615136876
+    },
+    "laya": {
+     "accuracy": 0.65,
+     "macro_f1": 0.6419437340153453
+    },
+    "gliner": {
+     "accuracy": 0.52,
+     "macro_f1": 0.39117199391172
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.73,
+     "macro_f1": 0.72997299729973
+    },
+    "julia": {
+     "accuracy": 0.63,
+     "macro_f1": 0.6189887756152817
+    },
+    "laya": {
+     "accuracy": 0.83,
+     "macro_f1": 0.8279178054458953
+    },
+    "gliner": {
+     "accuracy": 0.62,
+     "macro_f1": 0.5703301673450927
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "wikitoxic_threat": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.47,
+     "macro_f1": 0.3497730339835603
+    },
+    "julia": {
+     "accuracy": 0.47,
+     "macro_f1": 0.467390212038991
+    },
+    "laya": {
+     "accuracy": 0.91,
+     "macro_f1": 0.90999099909991
+    },
+    "gliner": {
+     "accuracy": 0.53,
+     "macro_f1": 0.39673982800667434
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.73,
+     "macro_f1": 0.7253585596582239
+    },
+    "julia": {
+     "accuracy": 0.55,
+     "macro_f1": 0.54226426609704
+    },
+    "laya": {
+     "accuracy": 0.77,
+     "macro_f1": 0.7613860358958398
+    },
+    "gliner": {
+     "accuracy": 0.52,
+     "macro_f1": 0.37629937629937626
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  },
+  "wikitoxic_toxicaggregated": {
+   "labels": 2,
+   "in_ines_ft_mixture": true,
+   "A": {
+    "ines": {
+     "accuracy": 0.33,
+     "macro_f1": 0.30490714804440294
+    },
+    "julia": {
+     "accuracy": 0.57,
+     "macro_f1": 0.5501621508525996
+    },
+    "laya": {
+     "accuracy": 0.7,
+     "macro_f1": 0.6921182266009853
+    },
+    "gliner": {
+     "accuracy": 0.62,
+     "macro_f1": 0.5766488413547237
+    }
+   },
+   "B": {
+    "ines": {
+     "accuracy": 0.77,
+     "macro_f1": 0.7681217864704104
+    },
+    "julia": {
+     "accuracy": 0.57,
+     "macro_f1": 0.557203171661003
+    },
+    "laya": {
+     "accuracy": 0.87,
+     "macro_f1": 0.8677652324280338
+    },
+    "gliner": {
+     "accuracy": 0.62,
+     "macro_f1": 0.5766488413547237
+    }
+   },
+   "trackA_strategy": {
+    "ines": "single request",
+    "julia": "single request",
+    "laya": "single request",
+    "gliner": "single request"
+   }
+  }
+ },
+ "global": {
+  "all 22": {
+   "A": {
+    "ines": 0.4739229278426516,
+    "julia": 0.5245951186849342,
+    "laya": 0.5301139532382007,
+    "gliner": 0.536710369477031
+   },
+   "B": {
+    "ines": 0.45679397195871585,
+    "julia": 0.3572179797391928,
+    "laya": 0.6294614568454058,
+    "gliner": 0.44952051094233986
+   }
+  },
+  "2 labels (13)": {
+   "A": {
+    "ines": 0.561538769269558,
+    "julia": 0.5825291987072047,
+    "laya": 0.7219571619118916,
+    "gliner": 0.627447823733444
+   },
+   "B": {
+    "ines": 0.6295911198690077,
+    "julia": 0.5172849909267092,
+    "laya": 0.7758169828614861,
+    "gliner": 0.5718255850749258
+   }
+  },
+  "3-20 labels (4)": {
+   "A": {
+    "ines": 0.48884952899904605,
+    "julia": 0.6335545038498116,
+    "laya": 0.5593357296978444,
+    "gliner": 0.5412491344877212
+   },
+   "B": {
+    "ines": 0.28323609763701546,
+    "julia": 0.23521866463768246,
+    "laya": 0.6478880971986014,
+    "gliner": 0.3784716661760502
+   }
+  },
+  "21-72 labels (5)": {
+   "A": {
+    "ines": 0.23418045920757943,
+    "julia": 0.28679900249512835,
+    "laya": 0.00794418951888898,
+    "gliner": 0.29716197640180464
+   },
+   "B": {
+    "ines": 0.14636768684931714,
+    "julia": 0.03864320273285837,
+    "laya": 0.23419577692104018,
+    "gliner": 0.18836639401064836
+   }
+  },
+  "absent from Ines-1 decision FT mixture (15)": {
+   "A": {
+    "ines": 0.4858805497724014,
+    "julia": 0.5360250230979545,
+    "laya": 0.5154640164128853,
+    "gliner": 0.5280778437879452
+   },
+   "B": {
+    "ines": 0.4094576296280325,
+    "julia": 0.3319018448171922,
+    "laya": 0.6114586574566909,
+    "gliner": 0.4284732740710553
+   }
+  },
+  "families in Ines-1 decision FT mixture (7)": {
+   "A": {
+    "ines": 0.44829945227890217,
+    "julia": 0.500102466371319,
+    "laya": 0.5615066750067336,
+    "gliner": 0.5552086388107859
+   },
+   "B": {
+    "ines": 0.5582289912387514,
+    "julia": 0.411466840286337,
+    "laya": 0.6680388841069373,
+    "gliner": 0.4946217328093781
+   }
+  }
+ },
+ "bootstrap": {
+  "all 22": {
+   "A:julia - A:ines": {
+    "difference": 0.050672190842282465,
+    "ci95": [
+     -0.026197789339992814,
+     0.12574349835362167
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.05619102539554903,
+    "ci95": [
+     -0.03957741468351192,
+     0.16332715413710902
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.06278744163437923,
+    "ci95": [
+     0.014208626140706566,
+     0.10706083324209556
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.099575992219523,
+    "ci95": [
+     -0.14637550498902815,
+     -0.05017137416536893
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.17266748488669,
+    "ci95": [
+     0.11284488099527132,
+     0.23744000375671462
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": -0.007273461016375937,
+    "ci95": [
+     -0.06982555475121606,
+     0.05100715361862577
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": -0.017128955883935842,
+    "ci95": [
+     -0.09913170297625284,
+     0.07045906519324419
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.16737713894574127,
+    "ci95": [
+     -0.2550239493204071,
+     -0.08802480974576297
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.09934750360720516,
+    "ci95": [
+     0.0454206858231911,
+     0.14433272375703962
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.087189858534691,
+    "ci95": [
+     -0.13204415683257073,
+     -0.0406803117967699
+    ]
+   }
+  },
+  "2 labels (13)": {
+   "A:julia - A:ines": {
+    "difference": 0.020990429437646712,
+    "ci95": [
+     -0.06061145608979183,
+     0.10740818043124331
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.16041839264233365,
+    "ci95": [
+     0.059458802921318936,
+     0.27206775011759543
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.0659090544638859,
+    "ci95": [
+     0.004675393764386895,
+     0.12582906809147268
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.11230612894229851,
+    "ci95": [
+     -0.18080307502401038,
+     -0.042581067377537536
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.14622586299247847,
+    "ci95": [
+     0.09520841847771228,
+     0.20567775456268422
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": -0.057765534794081974,
+    "ci95": [
+     -0.1428996478267053,
+     0.029876283921421498
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": 0.06805235059944967,
+    "ci95": [
+     -0.04906521051519053,
+     0.19755199873107607
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.06524420778049556,
+    "ci95": [
+     -0.12548066475124145,
+     -0.012510733064128867
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.05385982094959447,
+    "ci95": [
+     -0.0009695761275055773,
+     0.1065780484194153
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.0556222386585182,
+    "ci95": [
+     -0.11618959302529945,
+     0.007571405517821818
+    ]
+   }
+  },
+  "3-20 labels (4)": {
+   "A:julia - A:ines": {
+    "difference": 0.14470497485076556,
+    "ci95": [
+     -0.12895039644581718,
+     0.42568245069405136
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.07048620069879838,
+    "ci95": [
+     -0.229964039114142,
+     0.3531563180395061
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.052399605488675075,
+    "ci95": [
+     -0.05581876515491288,
+     0.1600753585487263
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.048017432999332976,
+    "ci95": [
+     -0.16164512046595278,
+     0.05754769356829411
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.364651999561586,
+    "ci95": [
+     0.18329205393405798,
+     0.5540391760322673
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": 0.09523556853903474,
+    "ci95": [
+     0.01900107622902177,
+     0.17430955225269407
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": -0.2056134313620306,
+    "ci95": [
+     -0.3164880910714138,
+     -0.09244502441135767
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.3983358392121291,
+    "ci95": [
+     -0.5756158997267588,
+     -0.20827382157801752
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.08855236750075707,
+    "ci95": [
+     -0.008752159824462086,
+     0.1939655463264431
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.16277746831167095,
+    "ci95": [
+     -0.25637377988986315,
+     -0.07162803351218687
+    ]
+   }
+  },
+  "21-72 labels (5)": {
+   "A:julia - A:ines": {
+    "difference": 0.05261854328754896,
+    "ci95": [
+     -0.04301551177445001,
+     0.15765204279382006
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": -0.22623626968869046,
+    "ci95": [
+     -0.30032025667320744,
+     -0.12187817221877785
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.06298151719422522,
+    "ci95": [
+     -0.039063579916225874,
+     0.1689149673736492
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.10772448411645877,
+    "ci95": [
+     -0.15259487999276594,
+     -0.04920310152464395
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.0878280900717231,
+    "ci95": [
+     0.011100563313016499,
+     0.14839732102914302
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": 0.041998707161331236,
+    "ci95": [
+     -0.019529909671919345,
+     0.11057761360469363
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": -0.08781277235826232,
+    "ci95": [
+     -0.12624266008952723,
+     -0.04849955912024958
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.24815579976227004,
+    "ci95": [
+     -0.3988206525705068,
+     -0.09861247338677506
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.22625158740215126,
+    "ci95": [
+     0.12816497504348637,
+     0.286499258070288
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.1087955823911563,
+    "ci95": [
+     -0.1700071752364474,
+     -0.032048604061386245
+    ]
+   }
+  },
+  "absent from Ines-1 decision FT mixture (15)": {
+   "A:julia - A:ines": {
+    "difference": 0.050144473325553045,
+    "ci95": [
+     -0.028885054771255515,
+     0.1500566532935792
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.029583466640483884,
+    "ci95": [
+     -0.08075161791194013,
+     0.14421513826606108
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.04219729401554383,
+    "ci95": [
+     -0.006600902674686908,
+     0.09437438464034366
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.0775557848108404,
+    "ci95": [
+     -0.14064220126958463,
+     -0.013517268561773333
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.2020010278286585,
+    "ci95": [
+     0.1269579464717378,
+     0.2900629696566373
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": 0.019015644443022794,
+    "ci95": [
+     -0.03174972794352044,
+     0.07223435697834166
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": -0.0764229201443689,
+    "ci95": [
+     -0.15017289217507543,
+     -0.009184764537925973
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.20412317828076224,
+    "ci95": [
+     -0.2998785466263466,
+     -0.11592881332069362
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.09599464104380576,
+    "ci95": [
+     0.0459047217540702,
+     0.1442013728576654
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.09960456971688988,
+    "ci95": [
+     -0.14649500828750583,
+     -0.04960181354566028
+    ]
+   }
+  },
+  "families in Ines-1 decision FT mixture (7)": {
+   "A:julia - A:ines": {
+    "difference": 0.051803014092416944,
+    "ci95": [
+     -0.10019425716510703,
+     0.1778425425557031
+    ]
+   },
+   "A:laya - A:ines": {
+    "difference": 0.11320722272783149,
+    "ci95": [
+     -0.12479087223953271,
+     0.3295312555301822
+    ]
+   },
+   "A:gliner - A:ines": {
+    "difference": 0.10690918653188367,
+    "ci95": [
+     0.019280206335977032,
+     0.19106643449909544
+    ]
+   },
+   "B:julia - B:ines": {
+    "difference": -0.14676215095241435,
+    "ci95": [
+     -0.1986636251903599,
+     -0.08740265140015356
+    ]
+   },
+   "B:laya - B:ines": {
+    "difference": 0.10980989286818602,
+    "ci95": [
+     0.03243625252632493,
+     0.19414880936010162
+    ]
+   },
+   "B:gliner - B:ines": {
+    "difference": -0.06360725842937322,
+    "ci95": [
+     -0.21886027228587646,
+     0.08094216222892091
+    ]
+   },
+   "B:ines - A:ines": {
+    "difference": 0.10992953895984924,
+    "ci95": [
+     -0.07562649281899102,
+     0.2927199100383737
+    ]
+   },
+   "B:julia - A:julia": {
+    "difference": -0.08863562608498207,
+    "ci95": [
+     -0.24495270619294315,
+     0.033542744774361914
+    ]
+   },
+   "B:laya - A:laya": {
+    "difference": 0.10653220910020378,
+    "ci95": [
+     -0.015590406007179902,
+     0.20797413933948058
+    ]
+   },
+   "B:gliner - A:gliner": {
+    "difference": -0.06058690600140765,
+    "ci95": [
+     -0.14995189706107254,
+     0.032545729267887305
+    ]
+   }
+  }
+ },
+ "calibration": {
+  "native (Track A, full native distributions, <=20 labels, 17 datasets)": {
+   "ines": {
+    "brier": 0.5014338202978408,
+    "nll": 0.8000206904458703,
+    "ece": 0.1270706365126021,
+    "coverage_at_error_budget": 0.08941176470588236,
+    "mean_confidence": 0.6118652016888647,
+    "datasets": 17
+   },
+   "julia": {
+    "brier": 0.645280763707045,
+    "nll": 1.6618435294062641,
+    "ece": 0.2983721069531266,
+    "coverage_at_error_budget": 0.1288235294117647,
+    "mean_confidence": 0.8941619627630315,
+    "datasets": 17
+   },
+   "laya": {
+    "brier": 0.4006447652764705,
+    "nll": 0.6925248427789752,
+    "ece": 0.09332229411764704,
+    "coverage_at_error_budget": 0.25588235294117645,
+    "mean_confidence": 0.6874152352941176,
+    "datasets": 17
+   },
+   "gliner": {
+    "brier": 0.48542838104085484,
+    "nll": 0.8599173300405782,
+    "ece": 0.1585389382487745,
+    "coverage_at_error_budget": 0.19705882352941173,
+    "mean_confidence": 0.7310719386879542,
+    "datasets": 17
+   }
+  },
+  "derived (Track B softmax of log-odds, same 17 datasets)": {
+   "ines": {
+    "brier": 0.5266089613872135,
+    "nll": 0.8626628971730973,
+    "ece": 0.11876053221277025,
+    "coverage_at_error_budget": 0.1335294117647059,
+    "mean_confidence": 0.4924301710578631,
+    "datasets": 17
+   },
+   "julia": {
+    "brier": 0.7685502933200778,
+    "nll": 1.5719702226511314,
+    "ece": 0.31343832319933446,
+    "coverage_at_error_budget": 0.014117647058823528,
+    "mean_confidence": 0.7770917395785171,
+    "datasets": 17
+   },
+   "laya": {
+    "brier": 0.35941673886508035,
+    "nll": 0.6807608663005745,
+    "ece": 0.12656927310814167,
+    "coverage_at_error_budget": 0.38176470588235295,
+    "mean_confidence": 0.7694262437653563,
+    "datasets": 17
+   },
+   "gliner": {
+    "brier": 0.5872442915941616,
+    "nll": 1.0308428769681912,
+    "ece": 0.2029575594285007,
+    "coverage_at_error_budget": 0.03705882352941177,
+    "mean_confidence": 0.7019368747032854,
+    "datasets": 17
+   }
+  },
+  "derived (Track B softmax of log-odds, all 22 datasets)": {
+   "ines": {
+    "brier": 0.6231953131399431,
+    "nll": 1.4522999819886815,
+    "ece": 0.12354853986128157,
+    "coverage_at_error_budget": 0.10590909090909091,
+    "mean_confidence": 0.39202771650296064,
+    "datasets": 22
+   },
+   "julia": {
+    "brier": 0.8554517402951244,
+    "nll": 2.468274766821923,
+    "ece": 0.31679964418123413,
+    "coverage_at_error_budget": 0.011363636363636364,
+    "mean_confidence": 0.687741939307485,
+    "datasets": 22
+   },
+   "laya": {
+    "brier": 0.47884708479764443,
+    "nll": 1.203588850522752,
+    "ece": 0.12330875423821963,
+    "coverage_at_error_budget": 0.29681818181818187,
+    "mean_confidence": 0.6690071844208508,
+    "datasets": 22
+   },
+   "gliner": {
+    "brier": 0.6643794367941797,
+    "nll": 1.5673671850720794,
+    "ece": 0.18732672413963056,
+    "coverage_at_error_budget": 0.031363636363636364,
+    "mean_confidence": 0.5655688289154754,
+    "datasets": 22
+   }
+  }
+ },
+ "ines_ablation": {
+  "agnews": {
+   "labels": 4,
+   "accuracy": {
+    "native(<=26)": 0.59,
+    "knockout-20": null,
+    "knockout-2": 0.64,
+    "one-vs-rest": 0.42
+   },
+   "macro_f1": {
+    "native(<=26)": 0.5843451916892106,
+    "knockout-20": null,
+    "knockout-2": 0.639691081949832,
+    "one-vs-rest": 0.32652938961350175
+   }
+  },
+  "emotiondair": {
+   "labels": 6,
+   "accuracy": {
+    "native(<=26)": 0.43,
+    "knockout-20": null,
+    "knockout-2": 0.43,
+    "one-vs-rest": 0.34
+   },
+   "macro_f1": {
+    "native(<=26)": 0.39042230395613853,
+    "knockout-20": null,
+    "knockout-2": 0.39217171717171717,
+    "one-vs-rest": 0.29478472669962036
+   }
+  },
+  "banking77": {
+   "labels": 72,
+   "accuracy": {
+    "native(<=26)": null,
+    "knockout-20": 0.39,
+    "knockout-2": 0.45,
+    "one-vs-rest": 0.29
+   },
+   "macro_f1": {
+    "native(<=26)": null,
+    "knockout-20": 0.34265873015873016,
+    "knockout-2": 0.4010912698412698,
+    "one-vs-rest": 0.2234898589065256
+   }
+  },
+  "financialphrasebank": {
+   "labels": 3,
+   "accuracy": {
+    "native(<=26)": 0.51,
+    "knockout-20": null,
+    "knockout-2": 0.44,
+    "one-vs-rest": 0.45
+   },
+   "macro_f1": {
+    "native(<=26)": 0.48911853494169083,
+    "knockout-20": null,
+    "knockout-2": 0.3744570258331727,
+    "one-vs-rest": 0.388558378810103
+   }
+  },
+  "empathetic": {
+   "labels": 32,
+   "accuracy": {
+    "native(<=26)": null,
+    "knockout-20": 0.25,
+    "knockout-2": 0.16,
+    "one-vs-rest": 0.18
+   },
+   "macro_f1": {
+    "native(<=26)": null,
+    "knockout-20": 0.1909003551273288,
+    "knockout-2": 0.10230728593731689,
+    "one-vs-rest": 0.14416745535166586
+   }
+  },
+  "massive": {
+   "labels": 59,
+   "accuracy": {
+    "native(<=26)": null,
+    "knockout-20": 0.35,
+    "knockout-2": 0.41,
+    "one-vs-rest": 0.2
+   },
+   "macro_f1": {
+    "native(<=26)": null,
+    "knockout-20": 0.2502433536331841,
+    "knockout-2": 0.3015873015873015,
+    "one-vs-rest": 0.15282852740479855
+   }
+  },
+  "yahootopics": {
+   "labels": 10,
+   "accuracy": {
+    "native(<=26)": 0.5,
+    "knockout-20": null,
+    "knockout-2": 0.57,
+    "one-vs-rest": 0.18
+   },
+   "macro_f1": {
+    "native(<=26)": 0.4915120854091442,
+    "knockout-20": null,
+    "knockout-2": 0.5584408865987813,
+    "one-vs-rest": 0.12307189542483658
+   }
+  },
+  "manifesto": {
+   "labels": 56,
+   "accuracy": {
+    "native(<=26)": null,
+    "knockout-20": 0.11,
+    "knockout-2": 0.18,
+    "one-vs-rest": 0.05
+   },
+   "macro_f1": {
+    "native(<=26)": null,
+    "knockout-20": 0.07504578754578754,
+    "knockout-2": 0.14917929292929294,
+    "one-vs-rest": 0.032902735562310034
+   }
+  },
+  "capsotu": {
+   "labels": 21,
+   "accuracy": {
+    "native(<=26)": 0.36,
+    "knockout-20": 0.37,
+    "knockout-2": 0.39,
+    "one-vs-rest": 0.21
+   },
+   "macro_f1": {
+    "native(<=26)": 0.3120540695728665,
+    "knockout-20": 0.3317682178584434,
+    "knockout-2": 0.3739835503810659,
+    "one-vs-rest": 0.17844985702128557
+   }
+  }
+ },
+ "anchors": {
+  "GLiNER2.5 Track A vs published pilot (accuracy)": {
+   "agnews": {
+    "ours": 0.7,
+    "published": 0.7
+   },
+   "emotiondair": {
+    "ours": 0.44,
+    "published": 0.44
+   },
+   "banking77": {
+    "ours": 0.61,
+    "published": 0.61
+   }
+  },
+  "Julia-1 Track A vs its model card (accuracy)": {
+   "agnews": {
+    "ours": 0.94,
+    "card": 0.94,
+    "ours_strategy": "single request"
+   },
+   "emotiondair": {
+    "ours": 0.86,
+    "card": 0.86,
+    "ours_strategy": "single request"
+   },
+   "banking77": {
+    "ours": 0.63,
+    "card": 0.64,
+    "ours_strategy": "Router(width=20, survivors=2)"
+   }
+  }
+ },
+ "ovr_ties": {
+  "ines": {
+   "examples_with_tie_at_max": 268,
+   "n": 2200
+  },
+  "julia": {
+   "examples_with_tie_at_max": 4,
+   "n": 2200
+  },
+  "laya": {
+   "examples_with_tie_at_max": 11,
+   "n": 2200
+  },
+  "gliner": {
+   "examples_with_tie_at_max": 0,
+   "n": 2200
+  }
+ }
+}

training/code/release/build_release.py

@@ -194,6 +194,8 @@ def main():
             shutil.copy(p, ev / "items" / p.name)
     for p in sorted((a.audit / "fullcase").glob("v3-full_*__*.jsonl")):
         shutil.copy(p, ev / "items" / p.name)
+    if (a.audit / "btzsc22").is_dir():  # external evaluation package (its own REPORT, PROTOCOL, SHA256SUMS)
+        shutil.copytree(a.audit / "btzsc22", ev / "btzsc22")
 
     # guard: no internal path, user, host or address may leave in the bundle
     leaks = []