Endikavi commited on
Commit
2e965de
·
verified ·
1 Parent(s): ed926f1

Add architecture diagram (assets/architecture.png + SVG source) and README Architecture section

Browse files
Files changed (5) hide show
  1. .gitattributes +1 -0
  2. README.md +13 -2
  3. SHA256SUMS +4 -2
  4. assets/architecture.png +3 -0
  5. assets/architecture.svg +249 -0
.gitattributes CHANGED
@@ -3,3 +3,4 @@
3
  *.pt filter=lfs diff=lfs merge=lfs -text
4
  tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
5
  *.pdf filter=lfs diff=lfs merge=lfs -text
 
 
3
  *.pt filter=lfs diff=lfs merge=lfs -text
4
  tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text
5
  *.pdf filter=lfs diff=lfs merge=lfs -text
6
+ *.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -33,7 +33,7 @@ datasets:
33
 
34
  > **Licences — what covers what.**
35
  > - **Apache-2.0** ([LICENSE](LICENSE), [NOTICE](NOTICE)): the model weights (`model.safetensors`), `config.json` and
36
- > `tokenizer/`; the documentation (`README.md` and the other `.md` files, `paper/`, `tasksource_license_audit.csv`); the
37
  > training records (`training/*.json`); the model's own evaluation outputs in `eval/`; `SHA256SUMS`.
38
  > - **MIT** ([LICENSE-CODE](LICENSE-CODE)): the code — `mini_v41/`, `mini_v41_jev/`, `scripts/`, `examples/`,
39
  > `training/code/`, `eval/btzsc22/code/` — and the environment and build files: `Dockerfile`, `.dockerignore`, `requirements.txt`,
@@ -54,6 +54,17 @@ full-case (multi-question) protocol.**
54
  Development codename: mini-v41 / mini-v41-Decisions. The Python packages keep that name (`mini_v41`,
55
  `mini_v41_jev`), and so do the provenance records in `config.json` and `training/`.
56
 
 
 
 
 
 
 
 
 
 
 
 
57
  ## Supported decision interface
58
 
59
  - **One typed question per model prompt.** The answer is the softmax over the option letters (at most 26) at the
@@ -358,7 +369,7 @@ manifests, data audit, scripts), `eval/` (reports, per-item outputs, speed proto
358
  `requirements.txt`, `requirements.lock`, `Dockerfile`, [CHANGELOG_CLAIMS.md](CHANGELOG_CLAIMS.md),
359
  [THIRD_PARTY_DATA_NOTICE.md](THIRD_PARTY_DATA_NOTICE.md), [LICENSING_NOTES.md](LICENSING_NOTES.md),
360
  [TASKSOURCE_LICENSE_AUDIT.md](TASKSOURCE_LICENSE_AUDIT.md) + `tasksource_license_audit.csv`,
361
- `paper/` (technical report: PDF + LaTeX sources), [LICENSE](LICENSE) (Apache-2.0), [NOTICE](NOTICE), [LICENSE-CODE](LICENSE-CODE) (MIT), `SHA256SUMS`.
362
 
363
  ## Paper
364
 
 
33
 
34
  > **Licences — what covers what.**
35
  > - **Apache-2.0** ([LICENSE](LICENSE), [NOTICE](NOTICE)): the model weights (`model.safetensors`), `config.json` and
36
+ > `tokenizer/`; the documentation (`README.md` and the other `.md` files, `paper/`, `assets/`, `tasksource_license_audit.csv`); the
37
  > training records (`training/*.json`); the model's own evaluation outputs in `eval/`; `SHA256SUMS`.
38
  > - **MIT** ([LICENSE-CODE](LICENSE-CODE)): the code — `mini_v41/`, `mini_v41_jev/`, `scripts/`, `examples/`,
39
  > `training/code/`, `eval/btzsc22/code/` — and the environment and build files: `Dockerfile`, `.dockerignore`, `requirements.txt`,
 
54
  Development codename: mini-v41 / mini-v41-Decisions. The Python packages keep that name (`mini_v41`,
55
  `mini_v41_jev`), and so do the provenance records in `config.json` and `training/`.
56
 
57
+ ## Architecture
58
+
59
+ ![Ines-1 architecture](assets/architecture.png)
60
+
61
+ Causal encoder–decoder (CED) mixture-of-experts with Engram n-gram memory: 8 encoder layers with full causal attention,
62
+ then 8 decoder layers that attend, in one softmax, to their own sliding-window K/V (128 tokens) and to a global K/V
63
+ projected once from the encoder output and shared by every decoder layer. All 16 layers use a MoE FFN (16 routed experts,
64
+ top-2, plus 1 shared expert, SwiGLU 1536); Engram is added to the residual before encoder layers 1 and 5. Context is
65
+ 2,048 tokens: raw tokens reach every position through the encoder → global K/V path, while features computed inside the
66
+ decoder travel 128 + 7·127 = 1,017 positions. Vector source: [assets/architecture.svg](assets/architecture.svg).
67
+
68
  ## Supported decision interface
69
 
70
  - **One typed question per model prompt.** The answer is the softmax over the option letters (at most 26) at the
 
369
  `requirements.txt`, `requirements.lock`, `Dockerfile`, [CHANGELOG_CLAIMS.md](CHANGELOG_CLAIMS.md),
370
  [THIRD_PARTY_DATA_NOTICE.md](THIRD_PARTY_DATA_NOTICE.md), [LICENSING_NOTES.md](LICENSING_NOTES.md),
371
  [TASKSOURCE_LICENSE_AUDIT.md](TASKSOURCE_LICENSE_AUDIT.md) + `tasksource_license_audit.csv`,
372
+ `assets/` (architecture diagram: PNG + SVG source), `paper/` (technical report: PDF + LaTeX sources), [LICENSE](LICENSE) (Apache-2.0), [NOTICE](NOTICE), [LICENSE-CODE](LICENSE-CODE) (MIT), `SHA256SUMS`.
373
 
374
  ## Paper
375
 
SHA256SUMS CHANGED
@@ -1,5 +1,5 @@
1
  97ad8982da911d928c3423f164a38f80608b98289ac4f4b3f292d90ae0ef2262 .dockerignore
2
- 026b23285c5d5e2b117ee5065229bf984d6ae6fad6b21ee1f04c934b7cad1d57 .gitattributes
3
  862263fa1f46c20f0d1e4dac5ffcc75abd55c08211b2c3864c5f8764b9d87793 .gitignore
4
  4463cc803842f7fe2406ce4e15039ff73842826d0f0c222f6857f73f8da7f54a CHANGELOG_CLAIMS.md
5
  e4a1aeafc1f6c01f8fff69aef8a2518631428f3ec96db4a9a56d7bb296cce95f Dockerfile
@@ -8,9 +8,11 @@ cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
8
  6a7ddcad077839a44697c005ea7e6f455caecaa8054412913108d6e21a1c9dbe LICENSE-CODE
9
  121b31536257bd7147d9c9b054f6fb4e2bb68734024415c081ac0b50c7cf48f1 LICENSING_NOTES.md
10
  1faada60db1ccfc073ae063fc474d9f75cba0c9d2683d03ad66b2074efeb166a NOTICE
11
- d74ebe08f28c4f95dbf275fa9ff015c4f6ca7aacf59e3f65add1e15c8572de3c README.md
12
  6ec8a2bb55f9331d77b9adbeae7fd71eeb5614a988cd21eee18c7bde775df5ba TASKSOURCE_LICENSE_AUDIT.md
13
  1a4df8f10bd30b01e6acc19ef2d935faee5485f0c25452ff64ba6c8598fabd97 THIRD_PARTY_DATA_NOTICE.md
 
 
14
  757003b29ec1d935f236c06ea1e39cf9362c43de358f8b3df74326f8bfdbd40d config.json
15
  8f296d2c1dbe85d285bb45cd80d841e733f84168ff130daf825933917684993b eval/btzsc22/PROTOCOL.md
16
  5cc7f2cd08668abe6ac5dd5455e61bc2315e556e26c45906150426810fd49455 eval/btzsc22/REPORT.md
 
1
  97ad8982da911d928c3423f164a38f80608b98289ac4f4b3f292d90ae0ef2262 .dockerignore
2
+ f958f33c67320596595c21e8e540481a03456fac85586dbf26f0542d02684f82 .gitattributes
3
  862263fa1f46c20f0d1e4dac5ffcc75abd55c08211b2c3864c5f8764b9d87793 .gitignore
4
  4463cc803842f7fe2406ce4e15039ff73842826d0f0c222f6857f73f8da7f54a CHANGELOG_CLAIMS.md
5
  e4a1aeafc1f6c01f8fff69aef8a2518631428f3ec96db4a9a56d7bb296cce95f Dockerfile
 
8
  6a7ddcad077839a44697c005ea7e6f455caecaa8054412913108d6e21a1c9dbe LICENSE-CODE
9
  121b31536257bd7147d9c9b054f6fb4e2bb68734024415c081ac0b50c7cf48f1 LICENSING_NOTES.md
10
  1faada60db1ccfc073ae063fc474d9f75cba0c9d2683d03ad66b2074efeb166a NOTICE
11
+ 8b3d5abc94f861cd0c3622a42f3403c7bfb33938009a629b05c88a9fad3f2d2a README.md
12
  6ec8a2bb55f9331d77b9adbeae7fd71eeb5614a988cd21eee18c7bde775df5ba TASKSOURCE_LICENSE_AUDIT.md
13
  1a4df8f10bd30b01e6acc19ef2d935faee5485f0c25452ff64ba6c8598fabd97 THIRD_PARTY_DATA_NOTICE.md
14
+ 9708385cf095b543e009b0dd6013aa9985ef68a5b05c5ebef4c14c6a7f97cfa2 assets/architecture.png
15
+ 1c641817fa2f3c2a64a2f52f48e65148aa907e4d2e3df68ccd21f6312666fc69 assets/architecture.svg
16
  757003b29ec1d935f236c06ea1e39cf9362c43de358f8b3df74326f8bfdbd40d config.json
17
  8f296d2c1dbe85d285bb45cd80d841e733f84168ff130daf825933917684993b eval/btzsc22/PROTOCOL.md
18
  5cc7f2cd08668abe6ac5dd5455e61bc2315e556e26c45906150426810fd49455 eval/btzsc22/REPORT.md
assets/architecture.png ADDED

Git LFS Details

  • SHA256: 9708385cf095b543e009b0dd6013aa9985ef68a5b05c5ebef4c14c6a7f97cfa2
  • Pointer size: 131 Bytes
  • Size of remote file: 427 kB
assets/architecture.svg ADDED