v1.1 errata wave-3: MODEL_CARD.md
Browse files- MODEL_CARD.md +2 -2
MODEL_CARD.md
CHANGED
|
@@ -33,7 +33,7 @@ Largha makes one structured decision per forward pass: given a `state` and a lis
|
|
| 33 |
|
| 34 |
- Base: `jhu-clsp/mmBERT-small` (upstream pre-training unchanged).
|
| 35 |
- Fine-tune data: `LocalLLaMA/typed-decisions` train split + flip-augmented option reorderings.
|
| 36 |
-
- Recipe (`rl_agent_config.json`): loss `ce+brier`, optimizer `adafactor`, lr_encoder 2e-5 / lr_head 1e-4, micro_batch 8, grad_accum 4, updates
|
| 37 |
- **No eval rows were used in training** (typed-decisions test split, JevBench held out).
|
| 38 |
|
| 39 |
## Inputs & outputs
|
|
@@ -65,7 +65,7 @@ The shipped column has **ECE 0.2519** (en). Calibration temperatures (0.8660205
|
|
| 65 |
## Bias, risks & limitations (honest disclosure)
|
| 66 |
|
| 67 |
- **JevBench gate not passed**: 0.5455 (126/231) vs the 58.4% acceptance gate. Published as measured; we never trained on the eval rows.
|
| 68 |
-
- **Option-order robustness is imperfect**: a 2.2% flip rate means about one answer change per ~46 reorders. It is *better* than Jev (~9%) and Laya out-of-domain (19.4%), but only a 1.5 pp gap vs Laya in-domain (3.7%). Never rely on order-invariance alone.
|
| 69 |
- **Not a safety oracle**: use it as a first-line gate with escalation (or a deterministic L0 rule layer such as phocinae-guard), never as the sole guard for destructive or safety-critical commands.
|
| 70 |
- **Chinese: in-mix, machine-translated-case evaluation**: zh evaluation runs on machine-translated English cases, and the training mix includes machine-translated Chinese (≈2,400 rows) plus a native-Chinese block (≈1,400 rows) — a fitted (not zero-shot) reading.
|
| 71 |
- **Context constraint**: the base encoder supports 8192 positions, but the decision head was trained with a 512-token default; long inputs degrade (16k/32k probes: 0.453 / 0.387).
|
|
|
|
| 33 |
|
| 34 |
- Base: `jhu-clsp/mmBERT-small` (upstream pre-training unchanged).
|
| 35 |
- Fine-tune data: `LocalLLaMA/typed-decisions` train split + flip-augmented option reorderings.
|
| 36 |
+
- Recipe (`rl_agent_config.json`): loss `ce+brier`, optimizer `adafactor`, lr_encoder 2e-5 / lr_head 1e-4, micro_batch 8, grad_accum 4, updates 300, ~0.044 h (local, tag `n13_r4`); `checkpoint_meta.json`: epoch 1/1, avg_loss 1.3689859.
|
| 37 |
- **No eval rows were used in training** (typed-decisions test split, JevBench held out).
|
| 38 |
|
| 39 |
## Inputs & outputs
|
|
|
|
| 65 |
## Bias, risks & limitations (honest disclosure)
|
| 66 |
|
| 67 |
- **JevBench gate not passed**: 0.5455 (126/231) vs the 58.4% acceptance gate. Published as measured; we never trained on the eval rows.
|
| 68 |
+
- **Option-order robustness is imperfect**: a 2.17% (flip400) / 2.83% (any of 3) flip rate means about one answer change per ~46 reorders. It is *better* than Jev (~9%) and Laya out-of-domain (19.4%), but only a 1.5 pp gap vs Laya in-domain (3.7%). Never rely on order-invariance alone.
|
| 69 |
- **Not a safety oracle**: use it as a first-line gate with escalation (or a deterministic L0 rule layer such as phocinae-guard), never as the sole guard for destructive or safety-critical commands.
|
| 70 |
- **Chinese: in-mix, machine-translated-case evaluation**: zh evaluation runs on machine-translated English cases, and the training mix includes machine-translated Chinese (≈2,400 rows) plus a native-Chinese block (≈1,400 rows) — a fitted (not zero-shot) reading.
|
| 71 |
- **Context constraint**: the base encoder supports 8192 positions, but the decision head was trained with a 512-token default; long inputs degrade (16k/32k probes: 0.453 / 0.387).
|