darkmaniac7 commited on
Commit
cd97a40
·
verified ·
1 Parent(s): 8c9c454

Card: consistent layout, file table, usage and metadata

Browse files

README.md only. Same licences and credits as before; adds a summary table, how to use it in TokForge, a file table with sizes from the Hub, and tidier metadata.

Files changed (1) hide show
  1. README.md +80 -61
README.md CHANGED
@@ -1,68 +1,96 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
3
  language:
4
- - en
5
  tags:
6
- - tokforge
7
- - mnn
8
- - android
9
- - mobile
10
- - speculative-decoding
11
- - qwen3
12
- - draft-model
13
- - text-generation
14
- - on-device
15
- pipeline_tag: text-generation
16
  inference: false
17
- base_model: Qwen/Qwen3-0.6B
18
- base_model_relation: finetune
19
  ---
20
 
21
- # Qwen3-0.6B-kl-baseline-20k-MNN
 
 
22
 
23
  Published by [TokForge](https://tokforge.ai) for on-device inference.
24
 
25
- Small `Qwen3-0.6B` draft model exported for **TokForge + MNN** speculative decoding on Android.
 
 
 
 
 
 
 
 
26
 
27
- This is a practical, mobile-oriented draft bundle rather than a standard Transformers checkpoint. It is intended to be paired with larger Qwen3 targets inside TokForge.
28
 
29
- ## Why this repo exists
 
 
 
30
 
31
- This model is one of the strongest lightweight draft candidates we have for mobile speculative decoding:
32
 
33
- - trained on `20K` teacher samples
34
- - KL-distilled from a `Qwen3-8B` teacher
35
- - exported as a ready-to-use `MNN` bundle
36
- - tuned for the classic `CPU draft + GPU/CPU target` TokForge flow
37
 
38
- ## Best-known use
 
 
39
 
40
- - Draft model backend: `CPU`
41
- - Draft threads: `2`
42
- - Draft predict length: `d=3`
43
- - Typical target pairing: `Qwen3-8B` in TokForge
44
 
45
- ## Benchmark notes
46
 
47
- We do not publish per-device numbers for this experimental variant. Results vary by device, target model, and workload. For measured results on the current default draft line, see [TokForge-AccelerationPack-Draft](https://huggingface.co/darkmaniac7/TokForge-AccelerationPack-Draft).
 
 
 
 
 
 
 
 
 
 
48
 
49
- Training acceptance (`alpha`) at the final logged epoch:
50
 
51
- - `0.7178`
 
52
 
53
- ## Included files
 
 
 
54
 
55
- - `llm.mnn`
56
- - `llm.mnn.weight`
57
- - `llm_config.json`
58
- - `config.json`
59
- - `config_cpu.json`
60
- - tokenizer files
61
- - ONNX export artifact for reference
62
 
63
- ## Usage
64
 
65
- This bundle is meant for **TokForge / MNN**, not standard HF Inference.
 
 
 
 
 
66
 
67
  Typical TokForge recipe:
68
 
@@ -91,33 +119,24 @@ Known-good draft-side config:
91
  }
92
  ```
93
 
94
- ## Notes
 
 
 
95
 
96
- - This is **not** a drop-in Transformers repo for HF Inference.
97
- - It is a runtime bundle for **TokForge / MNN**.
98
- - If you want the safest current mobile draft option, this is one of the best places to start.
99
 
100
- ## Limitations and Intended Use
101
 
102
- - Intended for speculative decoding with larger `Qwen3` targets inside TokForge.
103
- - Best current evidence is strongest on `Qwen3-8B`; smaller or differently-paired targets may behave differently.
104
  - Mobile results depend heavily on backend routing, prompt length, and device thermals.
 
105
  - This is a specialized runtime artifact, not a general-purpose pretrained release.
106
 
107
- ## Collection
108
-
109
- - [TokForge Mobile Draft Models](https://huggingface.co/collections/darkmaniac7/tokforge-mobile-draft-models-69c36153ea7084ce78329665)
110
-
111
- ## TokForge
112
-
113
- - **Website:** https://tokforge.ai
114
- - **Discord:** https://discord.gg/Acv3CBtfVm
115
- - **Google Play:** https://play.google.com/store/apps/details?id=dev.tokforge
116
- - **iOS TestFlight:** https://testflight.apple.com/join/jnufjzRr
117
-
118
- Runs on-device in the TokForge app.
119
 
120
- If you benchmark this on your own device, feel free to share results in Discord.
121
 
122
  ## Credits
123
 
 
1
  ---
2
  license: apache-2.0
3
+ base_model: Qwen/Qwen3-0.6B
4
+ base_model_relation: finetune
5
+ pipeline_tag: text-generation
6
+ library_name: mnn
7
  language:
8
+ - en
9
  tags:
10
+ - tokforge
11
+ - on-device
12
+ - mnn
13
+ - android
14
+ - mobile
15
+ - speculative-decoding
16
+ - qwen3
17
+ - draft-model
18
+ - text-generation
 
19
  inference: false
 
 
20
  ---
21
 
22
+ # Qwen3 0.6B KL baseline 20k draft (MNN)
23
+
24
+ A research draft: Qwen3 0.6B KL-distilled from Qwen3 8B on 20K teacher samples, as an MNN bundle for speculative decoding experiments.
25
 
26
  Published by [TokForge](https://tokforge.ai) for on-device inference.
27
 
28
+ | | |
29
+ |---|---|
30
+ | **What it is** | Research draft (draft model for speculative decoding) |
31
+ | **Format** | MNN bundle, 4-bit weights (block 128) |
32
+ | **Runs on** | Android phones (CPU), MNN runtime; not offered in the TokForge app |
33
+ | **Phone class** | No catalog class; needs more free RAM than its 336 MB of weights |
34
+ | **Download** | 340 MB (9 files) |
35
+ | **Base model** | [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) |
36
+ | **License** | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
37
 
38
+ ## TokForge
39
 
40
+ - **Website:** https://tokforge.ai
41
+ - **Discord:** https://discord.gg/Acv3CBtfVm
42
+ - **Google Play:** https://play.google.com/store/apps/details?id=dev.tokforge
43
+ - **iOS TestFlight:** https://testflight.apple.com/join/jnufjzRr
44
 
45
+ Runs on-device in the TokForge app.
46
 
47
+ ## Use it in TokForge
 
 
 
48
 
49
+ 1. Install TokForge from [Google Play](https://play.google.com/store/apps/details?id=dev.tokforge).
50
+ 2. TokForge does not offer or download this research draft. For speculative decoding with Qwen3 chat models, open **Settings > Speed & engine > Models & Modules** and download **Acceleration Pack: Qwen3-0.6B 40k** ([TokForge-AccelerationPack-Draft](https://huggingface.co/darkmaniac7/TokForge-AccelerationPack-Draft)).
51
+ 3. To try this draft yourself, run it with MNN's speculative decoding settings below.
52
 
53
+ Collections: [TokForge Research Drafts](https://huggingface.co/collections/darkmaniac7/tokforge-research-drafts-6af3fde6e8cfcf8046db20b6).
 
 
 
54
 
55
+ ## Files
56
 
57
+ | File | Size | What it is |
58
+ |---|---:|---|
59
+ | `config.json` | 210 B | MNN runtime settings |
60
+ | `config_cpu.json` | 167 B | Draft runtime settings (CPU, 2 threads, greedy) |
61
+ | `export_args.json` | 1.11 KB | Export settings used for the conversion |
62
+ | `llm.mnn` | 504 KB | Model graph |
63
+ | `llm.mnn.json` | 1.01 MB | Model graph as JSON (reference copy) |
64
+ | `llm.mnn.weight` | 336 MB | Model weights |
65
+ | `llm_config.json` | 4.66 KB | Model settings and chat template |
66
+ | `onnx/llm.onnx` | 933 KB | ONNX graph from the export (reference) |
67
+ | `tokenizer.txt` | 1.61 MB | Tokenizer |
68
 
69
+ ## About the model
70
 
71
+ Small `Qwen3-0.6B` draft model exported for **TokForge + MNN** speculative decoding on Android. It is meant to be
72
+ paired with larger Qwen3 targets; it is not a chat model on its own.
73
 
74
+ - trained on `20K` teacher samples
75
+ - KL-distilled from a `Qwen3-8B` teacher
76
+ - exported as a ready-to-use `MNN` bundle
77
+ - tuned for the classic `CPU draft + GPU/CPU target` TokForge flow
78
 
79
+ | | |
80
+ |---|---|
81
+ | **Architecture** | Qwen3 (dense, full attention) |
82
+ | **Parameters** | 0.6B (4-bit quantized) |
83
+ | **Quantization** | W4A16 (4-bit weights, block size 128) |
84
+ | **Teacher** | [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) |
 
85
 
86
+ ## Recommended settings
87
 
88
+ | Setting | Value |
89
+ |---------|-------|
90
+ | Draft model backend | CPU |
91
+ | Draft threads | 2 |
92
+ | Draft predict length | d=3 |
93
+ | Typical target pairing | Qwen3-8B |
94
 
95
  Typical TokForge recipe:
96
 
 
119
  }
120
  ```
121
 
122
+ ## How it was made
123
+
124
+ KL distillation of `Qwen3-0.6B` toward a `Qwen3-8B` teacher on `20K` teacher samples, then exported with MNN's
125
+ `llmexport` with 4-bit weights in blocks of 128 (`export_args.json` has the full settings).
126
 
127
+ Training acceptance (`alpha`) at the final logged epoch: `0.7178`.
 
 
128
 
129
+ ## Limitations
130
 
131
+ - Research draft: TokForge does not offer it, and we do not publish per-device numbers for it. Results vary by device, target model, and workload.
132
+ - Intended for speculative decoding with larger `Qwen3` targets.
133
  - Mobile results depend heavily on backend routing, prompt length, and device thermals.
134
+ - This is **not** a drop-in Transformers repo for HF Inference; it is a runtime bundle for **TokForge / MNN**.
135
  - This is a specialized runtime artifact, not a general-purpose pretrained release.
136
 
137
+ ## License
 
 
 
 
 
 
 
 
 
 
 
138
 
139
+ Apache License 2.0 ([text](https://www.apache.org/licenses/LICENSE-2.0)), the license of Qwen3 0.6B and of the Qwen3 8B teacher.
140
 
141
  ## Credits
142