--- license: cc-by-nc-4.0 language: - en - fr pipeline_tag: text-generation library_name: transformers tags: - pytorch - safetensors - tr-hash - mixture-of-experts - gqa - supervised-finetuning - full-parameter-finetuning - custom-code base_model: AETHORIA-AI/TR-HASH-MoE-200M-160B-Refinement datasets: - AETHORIA-AI/TR-HASH-MoE-200M-SFT-v2-300K --- # TR-HASH MoE 200M — 160B-source Full SFT (32K release) ## Paper **Deterministic multi-hash routing supports long-horizon training in a compact language model** [Preprint (DOI)](https://doi.org/10.21203/rs.3.rs-10788774/v1) · [Model weights](https://huggingface.co/AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT) · [Code](https://github.com/Complexity-ML/complexity-framework) Full-parameter instruction SFT of [`AETHORIA-AI/TR-HASH-MoE-200M-160B-Refinement`](https://huggingface.co/AETHORIA-AI/TR-HASH-MoE-200M-160B-Refinement) on the audited [`AETHORIA-AI/TR-HASH-MoE-200M-SFT-v2-300K`](https://huggingface.co/datasets/AETHORIA-AI/TR-HASH-MoE-200M-SFT-v2-300K). This release is **not LoRA or QLoRA**: all 201.2M model parameters were trainable. The root `model.safetensors` is the **epoch 3 / step 5,982** checkpoint, selected for the strongest full-PIQA result and lowest held-out SFT loss across the three epochs. > **Tokenizer:** use > [`AETHORIA-AI/TR-HASH-Tokenizer-32K`](https://huggingface.co/AETHORIA-AI/TR-HASH-Tokenizer-32K) > with this 32,000-token model. ## Results | Epoch | Step | Held-out SFT loss | SFT ppl | PIQA acc | PIQA acc_norm | |---:|---:|---:|---:|---:|---:| | 1 | 1,994 | 0.990943 | 2.69 | 67.90% | 68.93% | | 2 | 3,988 | 0.963912 | 2.62 | 67.85% | 68.82% | | 3 | 5,982 | 0.959617 | 2.61 | 68.01% | 69.10% | ### Released checkpoint benchmark panel | Benchmark | Split | Examples | Accuracy | Accuracy (length-normalized) | Evaluation backend | |---|---|---:|---:|---:|---| | PIQA | validation | 1,838 | **68.01%** | **69.10%** | PyTorch FP16, custom Triton | | ARC-Easy | test | 2,376 | **57.24%** | **52.86%** | MLX FP16 | | ARC-Challenge | test | 1,172 | **27.13%** | **29.01%** | MLX FP16 | | **ARC Combined (micro)** | test | **3,548** | **47.29%** | **44.98%** | Derived from both ARC test splits | | HellaSwag | validation | 10,042 | **33.21%** | **38.74%** | MLX FP16 | All benchmark evaluations use zero-shot causal continuation log-likelihood, no chat template and a maximum sequence length of 2,048. Accuracy selects the choice with the highest total continuation log-likelihood; the normalized metric selects by mean continuation log-likelihood per scored token. PIQA was evaluated from the native epoch-3 checkpoint. ARC and HellaSwag were evaluated from an FP16 MLX conversion of the same root F32 SafeTensors weights. ARC Combined is the micro-average over all 3,548 ARC-Easy and ARC-Challenge test examples, not the arithmetic mean of the two percentages. Machine-readable reports are published under `reports/sft-v2-300k/`. ### Compact-model comparison ![Combined ARC comparison from 124M to 774M parameters](assets/combined_arc_model_comparison.png) | Model | Parameters | PIQA acc | ARC-Easy acc | ARC-Challenge acc | **Combined ARC acc** | HellaSwag acc | |---|---:|---:|---:|---:|---:|---:| | **TR-HASH MoE 200M Full SFT** | **201.2M** | **68.01%** | **57.24%** | **27.13%** | **47.29%** | **33.21%** | | GPT-2 Large | 774M | — | 53.11% | 21.76% | 42.76% | — | | Pythia-410M | 410M | — | 52.02% | 21.42% | 41.91% | — | | GPT-2 Medium | 355M | — | 49.16% | 21.67% | 40.08% | — | | OPT-350M | 350M | — | 43.98% | 20.82% | 36.33% | — | | GPT-2 Small | 124M | 62.89% | 43.81% | 19.03% | ≈35.63% | 28.92% | | OPT-125M | 125M | 63.00% | 43.60% | 19.10% | ≈35.51% | 29.20% | | Pythia-160M | 160M | 62.73% | 43.52% | 18.77% | 35.34% | — | Combined ARC is weighted by the public test-set sizes (2,376 ARC-Easy and 1,172 ARC-Challenge examples). GPT-2 Small and OPT reference scores come from the [AMD-LLM lm-evaluation-harness comparison](https://github.com/AMD-AGI/AMD-LLM#evaluation); their combined values are approximate because the published component scores are rounded. Pythia-160M uses EleutherAI's [official zero-shot result](https://github.com/EleutherAI/pythia/blob/main/evals/pythia-v1/pythia-160m/zero-shot/160m_step143000.json). GPT-2 Medium, GPT-2 Large and Pythia-410M were evaluated locally in MLX FP16 with the same causal-choice evaluator as TR-HASH. OPT-350M used the identical prompt and scoring formula in PyTorch MPS FP16 because MLX does not implement the OPT architecture. The 124M–160M references were reported through lm-evaluation-harness, so that part of the comparison is informative rather than a claim of bit-identical evaluation runtimes. ## Training recipe | Setting | Value | |---|---| | Method | Full-parameter supervised fine-tuning | | Source | Refinement step 8,156 (about 162B prior token exposures) | | Dataset | 300,000 train / 3,000 held-out examples | | Tokenized corpus | 202,948,693 train tokens; no truncation | | Supervision | Final assistant turn only; prior assistant turns masked | | Epochs | 3 | | Context | 2,048 tokens | | Tokenizer | [`TR-HASH-Tokenizer-32K`](https://huggingface.co/AETHORIA-AI/TR-HASH-Tokenizer-32K); EOS `` (ID 0) | | Optimizer | AdamW, betas 0.9 / 0.95, weight decay 0.1 | | LR | 2e-5 peak, 3% warmup, continuous cosine decay | | Precision | BF16 training | | Root SafeTensors precision | float32 | | Kernels | Liger required; custom Triton enabled | ## Architecture and loading 201.2M parameters, 16 decoder layers, GQA (14 query heads / 2 KV heads), four stored deterministic token-ID-routed experts with top-2 activation, an always-on shared SwiGLU path and tied embeddings. The persisted multi-hash routing tables are part of the checkpoint. The repository includes an autonomous Transformers adapter. Load it with: ```python from transformers import AutoModelForCausalLM, AutoTokenizer repo = "AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT" tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True) ``` ## License The released model weights and repository code are licensed under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) as reproduced in [`LICENSE`](LICENSE). Commercial use is not permitted under this license. Source datasets retain their own licenses and terms; see the dataset manifest for the per-source audit. ## Citation ```bibtex @article{Peyriguere_2026, title = {Deterministic multi-hash routing supports long-horizon training in a compact language model}, author = {Peyriguere, Boris}, year = {2026}, month = aug, publisher = {Springer Science and Business Media LLC}, doi = {10.21203/rs.3.rs-10788774/v1}, url = {https://doi.org/10.21203/rs.3.rs-10788774/v1} } ```