Add architecture diagram (assets/architecture.png + SVG source) and README Architecture section
Browse files- .gitattributes +1 -0
- README.md +13 -2
- SHA256SUMS +4 -2
- assets/architecture.png +3 -0
- assets/architecture.svg +249 -0
.gitattributes
CHANGED
|
@@ -3,3 +3,4 @@
|
|
| 3 |
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 4 |
tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 5 |
*.pdf filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 3 |
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 4 |
tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
| 5 |
*.pdf filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -33,7 +33,7 @@ datasets:
|
|
| 33 |
|
| 34 |
> **Licences — what covers what.**
|
| 35 |
> - **Apache-2.0** ([LICENSE](LICENSE), [NOTICE](NOTICE)): the model weights (`model.safetensors`), `config.json` and
|
| 36 |
-
> `tokenizer/`; the documentation (`README.md` and the other `.md` files, `paper/`, `tasksource_license_audit.csv`); the
|
| 37 |
> training records (`training/*.json`); the model's own evaluation outputs in `eval/`; `SHA256SUMS`.
|
| 38 |
> - **MIT** ([LICENSE-CODE](LICENSE-CODE)): the code — `mini_v41/`, `mini_v41_jev/`, `scripts/`, `examples/`,
|
| 39 |
> `training/code/`, `eval/btzsc22/code/` — and the environment and build files: `Dockerfile`, `.dockerignore`, `requirements.txt`,
|
|
@@ -54,6 +54,17 @@ full-case (multi-question) protocol.**
|
|
| 54 |
Development codename: mini-v41 / mini-v41-Decisions. The Python packages keep that name (`mini_v41`,
|
| 55 |
`mini_v41_jev`), and so do the provenance records in `config.json` and `training/`.
|
| 56 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
## Supported decision interface
|
| 58 |
|
| 59 |
- **One typed question per model prompt.** The answer is the softmax over the option letters (at most 26) at the
|
|
@@ -358,7 +369,7 @@ manifests, data audit, scripts), `eval/` (reports, per-item outputs, speed proto
|
|
| 358 |
`requirements.txt`, `requirements.lock`, `Dockerfile`, [CHANGELOG_CLAIMS.md](CHANGELOG_CLAIMS.md),
|
| 359 |
[THIRD_PARTY_DATA_NOTICE.md](THIRD_PARTY_DATA_NOTICE.md), [LICENSING_NOTES.md](LICENSING_NOTES.md),
|
| 360 |
[TASKSOURCE_LICENSE_AUDIT.md](TASKSOURCE_LICENSE_AUDIT.md) + `tasksource_license_audit.csv`,
|
| 361 |
-
`paper/` (technical report: PDF + LaTeX sources), [LICENSE](LICENSE) (Apache-2.0), [NOTICE](NOTICE), [LICENSE-CODE](LICENSE-CODE) (MIT), `SHA256SUMS`.
|
| 362 |
|
| 363 |
## Paper
|
| 364 |
|
|
|
|
| 33 |
|
| 34 |
> **Licences — what covers what.**
|
| 35 |
> - **Apache-2.0** ([LICENSE](LICENSE), [NOTICE](NOTICE)): the model weights (`model.safetensors`), `config.json` and
|
| 36 |
+
> `tokenizer/`; the documentation (`README.md` and the other `.md` files, `paper/`, `assets/`, `tasksource_license_audit.csv`); the
|
| 37 |
> training records (`training/*.json`); the model's own evaluation outputs in `eval/`; `SHA256SUMS`.
|
| 38 |
> - **MIT** ([LICENSE-CODE](LICENSE-CODE)): the code — `mini_v41/`, `mini_v41_jev/`, `scripts/`, `examples/`,
|
| 39 |
> `training/code/`, `eval/btzsc22/code/` — and the environment and build files: `Dockerfile`, `.dockerignore`, `requirements.txt`,
|
|
|
|
| 54 |
Development codename: mini-v41 / mini-v41-Decisions. The Python packages keep that name (`mini_v41`,
|
| 55 |
`mini_v41_jev`), and so do the provenance records in `config.json` and `training/`.
|
| 56 |
|
| 57 |
+
## Architecture
|
| 58 |
+
|
| 59 |
+

|
| 60 |
+
|
| 61 |
+
Causal encoder–decoder (CED) mixture-of-experts with Engram n-gram memory: 8 encoder layers with full causal attention,
|
| 62 |
+
then 8 decoder layers that attend, in one softmax, to their own sliding-window K/V (128 tokens) and to a global K/V
|
| 63 |
+
projected once from the encoder output and shared by every decoder layer. All 16 layers use a MoE FFN (16 routed experts,
|
| 64 |
+
top-2, plus 1 shared expert, SwiGLU 1536); Engram is added to the residual before encoder layers 1 and 5. Context is
|
| 65 |
+
2,048 tokens: raw tokens reach every position through the encoder → global K/V path, while features computed inside the
|
| 66 |
+
decoder travel 128 + 7·127 = 1,017 positions. Vector source: [assets/architecture.svg](assets/architecture.svg).
|
| 67 |
+
|
| 68 |
## Supported decision interface
|
| 69 |
|
| 70 |
- **One typed question per model prompt.** The answer is the softmax over the option letters (at most 26) at the
|
|
|
|
| 369 |
`requirements.txt`, `requirements.lock`, `Dockerfile`, [CHANGELOG_CLAIMS.md](CHANGELOG_CLAIMS.md),
|
| 370 |
[THIRD_PARTY_DATA_NOTICE.md](THIRD_PARTY_DATA_NOTICE.md), [LICENSING_NOTES.md](LICENSING_NOTES.md),
|
| 371 |
[TASKSOURCE_LICENSE_AUDIT.md](TASKSOURCE_LICENSE_AUDIT.md) + `tasksource_license_audit.csv`,
|
| 372 |
+
`assets/` (architecture diagram: PNG + SVG source), `paper/` (technical report: PDF + LaTeX sources), [LICENSE](LICENSE) (Apache-2.0), [NOTICE](NOTICE), [LICENSE-CODE](LICENSE-CODE) (MIT), `SHA256SUMS`.
|
| 373 |
|
| 374 |
## Paper
|
| 375 |
|
SHA256SUMS
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
97ad8982da911d928c3423f164a38f80608b98289ac4f4b3f292d90ae0ef2262 .dockerignore
|
| 2 |
-
|
| 3 |
862263fa1f46c20f0d1e4dac5ffcc75abd55c08211b2c3864c5f8764b9d87793 .gitignore
|
| 4 |
4463cc803842f7fe2406ce4e15039ff73842826d0f0c222f6857f73f8da7f54a CHANGELOG_CLAIMS.md
|
| 5 |
e4a1aeafc1f6c01f8fff69aef8a2518631428f3ec96db4a9a56d7bb296cce95f Dockerfile
|
|
@@ -8,9 +8,11 @@ cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
|
| 8 |
6a7ddcad077839a44697c005ea7e6f455caecaa8054412913108d6e21a1c9dbe LICENSE-CODE
|
| 9 |
121b31536257bd7147d9c9b054f6fb4e2bb68734024415c081ac0b50c7cf48f1 LICENSING_NOTES.md
|
| 10 |
1faada60db1ccfc073ae063fc474d9f75cba0c9d2683d03ad66b2074efeb166a NOTICE
|
| 11 |
-
|
| 12 |
6ec8a2bb55f9331d77b9adbeae7fd71eeb5614a988cd21eee18c7bde775df5ba TASKSOURCE_LICENSE_AUDIT.md
|
| 13 |
1a4df8f10bd30b01e6acc19ef2d935faee5485f0c25452ff64ba6c8598fabd97 THIRD_PARTY_DATA_NOTICE.md
|
|
|
|
|
|
|
| 14 |
757003b29ec1d935f236c06ea1e39cf9362c43de358f8b3df74326f8bfdbd40d config.json
|
| 15 |
8f296d2c1dbe85d285bb45cd80d841e733f84168ff130daf825933917684993b eval/btzsc22/PROTOCOL.md
|
| 16 |
5cc7f2cd08668abe6ac5dd5455e61bc2315e556e26c45906150426810fd49455 eval/btzsc22/REPORT.md
|
|
|
|
| 1 |
97ad8982da911d928c3423f164a38f80608b98289ac4f4b3f292d90ae0ef2262 .dockerignore
|
| 2 |
+
f958f33c67320596595c21e8e540481a03456fac85586dbf26f0542d02684f82 .gitattributes
|
| 3 |
862263fa1f46c20f0d1e4dac5ffcc75abd55c08211b2c3864c5f8764b9d87793 .gitignore
|
| 4 |
4463cc803842f7fe2406ce4e15039ff73842826d0f0c222f6857f73f8da7f54a CHANGELOG_CLAIMS.md
|
| 5 |
e4a1aeafc1f6c01f8fff69aef8a2518631428f3ec96db4a9a56d7bb296cce95f Dockerfile
|
|
|
|
| 8 |
6a7ddcad077839a44697c005ea7e6f455caecaa8054412913108d6e21a1c9dbe LICENSE-CODE
|
| 9 |
121b31536257bd7147d9c9b054f6fb4e2bb68734024415c081ac0b50c7cf48f1 LICENSING_NOTES.md
|
| 10 |
1faada60db1ccfc073ae063fc474d9f75cba0c9d2683d03ad66b2074efeb166a NOTICE
|
| 11 |
+
8b3d5abc94f861cd0c3622a42f3403c7bfb33938009a629b05c88a9fad3f2d2a README.md
|
| 12 |
6ec8a2bb55f9331d77b9adbeae7fd71eeb5614a988cd21eee18c7bde775df5ba TASKSOURCE_LICENSE_AUDIT.md
|
| 13 |
1a4df8f10bd30b01e6acc19ef2d935faee5485f0c25452ff64ba6c8598fabd97 THIRD_PARTY_DATA_NOTICE.md
|
| 14 |
+
9708385cf095b543e009b0dd6013aa9985ef68a5b05c5ebef4c14c6a7f97cfa2 assets/architecture.png
|
| 15 |
+
1c641817fa2f3c2a64a2f52f48e65148aa907e4d2e3df68ccd21f6312666fc69 assets/architecture.svg
|
| 16 |
757003b29ec1d935f236c06ea1e39cf9362c43de358f8b3df74326f8bfdbd40d config.json
|
| 17 |
8f296d2c1dbe85d285bb45cd80d841e733f84168ff130daf825933917684993b eval/btzsc22/PROTOCOL.md
|
| 18 |
5cc7f2cd08668abe6ac5dd5455e61bc2315e556e26c45906150426810fd49455 eval/btzsc22/REPORT.md
|
assets/architecture.png
ADDED
|
Git LFS Details
|
assets/architecture.svg
ADDED
|
|