Add extended Combined ARC comparison

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/combined_arc_model_comparison.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -60,9 +60,15 @@ Machine-readable reports are published under `reports/sft-v2-300k/`.
 
 ### Compact-model comparison
 
+![Combined ARC comparison from 124M to 774M parameters](assets/combined_arc_model_comparison.png)
+
 | Model | Parameters | PIQA acc | ARC-Easy acc | ARC-Challenge acc | **Combined ARC acc** | HellaSwag acc |
 |---|---:|---:|---:|---:|---:|---:|
 | **TR-HASH MoE 200M Full SFT** | **201.2M** | **68.01%** | **57.24%** | **27.13%** | **47.29%** | **33.21%** |
+| GPT-2 Large | 774M | — | 53.11% | 21.76% | 42.76% | — |
+| Pythia-410M | 410M | — | 52.02% | 21.42% | 41.91% | — |
+| GPT-2 Medium | 355M | — | 49.16% | 21.67% | 40.08% | — |
+| OPT-350M | 350M | — | 43.98% | 20.82% | 36.33% | — |
 | GPT-2 Small | 124M | 62.89% | 43.81% | 19.03% | ≈35.63% | 28.92% |
 | OPT-125M | 125M | 63.00% | 43.60% | 19.10% | ≈35.51% | 29.20% |
 | Pythia-160M | 160M | 62.73% | 43.52% | 18.77% | 35.34% | — |
@@ -73,9 +79,12 @@ the [AMD-LLM lm-evaluation-harness comparison](https://github.com/AMD-AGI/AMD-LL
 their combined values are approximate because the published component scores
 are rounded. Pythia-160M uses EleutherAI's
 [official zero-shot result](https://github.com/EleutherAI/pythia/blob/main/evals/pythia-v1/pythia-160m/zero-shot/160m_step143000.json).
-The comparison is informative rather than a claim of bit-identical evaluation
-runtimes: TR-HASH was scored by the repository's causal-choice evaluator,
-whereas the references were reported through lm-evaluation-harness.
+GPT-2 Medium, GPT-2 Large and Pythia-410M were evaluated locally in MLX FP16
+with the same causal-choice evaluator as TR-HASH. OPT-350M used the identical
+prompt and scoring formula in PyTorch MPS FP16 because MLX does not implement
+the OPT architecture. The 124M–160M references were reported through
+lm-evaluation-harness, so that part of the comparison is informative rather
+than a claim of bit-identical evaluation runtimes.
 
 ## Training recipe
 

reports/sft-v2-300k/arc_combined_model_comparison.json

@@ -0,0 +1,75 @@
+{
+  "aggregation": "micro-average over 2376 ARC-Easy and 1172 ARC-Challenge test examples",
+  "metric": "raw causal-choice accuracy",
+  "models": [
+    {
+      "model": "AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT",
+      "parameters": 201194880,
+      "arc_easy_acc": 0.5723905723905723,
+      "arc_challenge_acc": 0.2713310580204778,
+      "arc_combined_acc": 0.4729425028184893,
+      "combined_correct": 1678,
+      "backend": "mlx-fp16"
+    },
+    {
+      "model": "openai-community/gpt2-large",
+      "parameters": 774030080,
+      "arc_easy_acc": 0.5311447811447811,
+      "arc_challenge_acc": 0.2175767918088737,
+      "arc_combined_acc": 0.427564825253664,
+      "combined_correct": 1517,
+      "backend": "mlx-fp16"
+    },
+    {
+      "model": "EleutherAI/pythia-410m",
+      "parameters": 405334016,
+      "arc_easy_acc": 0.5202020202020202,
+      "arc_challenge_acc": 0.21416382252559726,
+      "arc_combined_acc": 0.41910935738444194,
+      "combined_correct": 1487,
+      "backend": "mlx-fp16"
+    },
+    {
+      "model": "openai-community/gpt2-medium",
+      "parameters": 354823168,
+      "arc_easy_acc": 0.49158249158249157,
+      "arc_challenge_acc": 0.2167235494880546,
+      "arc_combined_acc": 0.4007891770011274,
+      "combined_correct": 1422,
+      "backend": "mlx-fp16"
+    },
+    {
+      "model": "facebook/opt-350m",
+      "parameters": 331196416,
+      "arc_easy_acc": 0.4398148148148148,
+      "arc_challenge_acc": 0.20819112627986347,
+      "arc_combined_acc": 0.36330326944757607,
+      "combined_correct": 1289,
+      "backend": "pytorch-mps-fp16"
+    },
+    {
+      "model": "openai-community/gpt2",
+      "parameters": 124439808,
+      "arc_easy_acc": 0.4381,
+      "arc_challenge_acc": 0.1903,
+      "arc_combined_acc_approx": 0.356257,
+      "source": "AMD-AGI/AMD-LLM published rounded lm-evaluation-harness scores"
+    },
+    {
+      "model": "facebook/opt-125m",
+      "parameters": 125239296,
+      "arc_easy_acc": 0.436,
+      "arc_challenge_acc": 0.191,
+      "arc_combined_acc_approx": 0.35513,
+      "source": "AMD-AGI/AMD-LLM published rounded lm-evaluation-harness scores"
+    },
+    {
+      "model": "EleutherAI/pythia-160m",
+      "parameters": 162322944,
+      "arc_easy_acc": 0.4351851851851852,
+      "arc_challenge_acc": 0.18771331058020477,
+      "arc_combined_acc": 0.35343855693348365,
+      "source": "EleutherAI official zero-shot result"
+    }
+  ]
+}