darkmaniac7's picture
Card: consistent layout, file table, usage and metadata
cd97a40 verified
|
Raw History Blame Contribute Delete
5.21 kB
metadata
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
base_model_relation: finetune
pipeline_tag: text-generation
library_name: mnn
language:
  - en
tags:
  - tokforge
  - on-device
  - mnn
  - android
  - mobile
  - speculative-decoding
  - qwen3
  - draft-model
  - text-generation
inference: false

Qwen3 0.6B KL baseline 20k draft (MNN)

A research draft: Qwen3 0.6B KL-distilled from Qwen3 8B on 20K teacher samples, as an MNN bundle for speculative decoding experiments.

Published by TokForge for on-device inference.

What it is Research draft (draft model for speculative decoding)
Format MNN bundle, 4-bit weights (block 128)
Runs on Android phones (CPU), MNN runtime; not offered in the TokForge app
Phone class No catalog class; needs more free RAM than its 336 MB of weights
Download 340 MB (9 files)
Base model Qwen/Qwen3-0.6B
License Apache 2.0

TokForge

Runs on-device in the TokForge app.

Use it in TokForge

  1. Install TokForge from Google Play.
  2. TokForge does not offer or download this research draft. For speculative decoding with Qwen3 chat models, open Settings > Speed & engine > Models & Modules and download Acceleration Pack: Qwen3-0.6B 40k (TokForge-AccelerationPack-Draft).
  3. To try this draft yourself, run it with MNN's speculative decoding settings below.

Collections: TokForge Research Drafts.

Files

File Size What it is
config.json 210 B MNN runtime settings
config_cpu.json 167 B Draft runtime settings (CPU, 2 threads, greedy)
export_args.json 1.11 KB Export settings used for the conversion
llm.mnn 504 KB Model graph
llm.mnn.json 1.01 MB Model graph as JSON (reference copy)
llm.mnn.weight 336 MB Model weights
llm_config.json 4.66 KB Model settings and chat template
onnx/llm.onnx 933 KB ONNX graph from the export (reference)
tokenizer.txt 1.61 MB Tokenizer

About the model

Small Qwen3-0.6B draft model exported for TokForge + MNN speculative decoding on Android. It is meant to be paired with larger Qwen3 targets; it is not a chat model on its own.

  • trained on 20K teacher samples
  • KL-distilled from a Qwen3-8B teacher
  • exported as a ready-to-use MNN bundle
  • tuned for the classic CPU draft + GPU/CPU target TokForge flow
Architecture Qwen3 (dense, full attention)
Parameters 0.6B (4-bit quantized)
Quantization W4A16 (4-bit weights, block size 128)
Teacher Qwen/Qwen3-8B

Recommended settings

Setting Value
Draft model backend CPU
Draft threads 2
Draft predict length d=3
Typical target pairing Qwen3-8B

Typical TokForge recipe:

{
  "backend_type": "opencl",
  "thread_num": 4,
  "precision": "low",
  "memory": "low",
  "sampler_type": "greedy",
  "speculative_type": "draftmodel",
  "draft_predict_length": 3,
  "draft_config_path": "/path/to/config_cpu.json"
}

Known-good draft-side config:

{
  "backend_type": "cpu",
  "thread_num": 2,
  "precision": "low",
  "memory": "low",
  "sampler_type": "greedy"
}

How it was made

KL distillation of Qwen3-0.6B toward a Qwen3-8B teacher on 20K teacher samples, then exported with MNN's llmexport with 4-bit weights in blocks of 128 (export_args.json has the full settings).

Training acceptance (alpha) at the final logged epoch: 0.7178.

Limitations

  • Research draft: TokForge does not offer it, and we do not publish per-device numbers for it. Results vary by device, target model, and workload.
  • Intended for speculative decoding with larger Qwen3 targets.
  • Mobile results depend heavily on backend routing, prompt length, and device thermals.
  • This is not a drop-in Transformers repo for HF Inference; it is a runtime bundle for TokForge / MNN.
  • This is a specialized runtime artifact, not a general-purpose pretrained release.

License

Apache License 2.0 (text), the license of Qwen3 0.6B and of the Qwen3 8B teacher.

Credits

  • Qwen3-0.6B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license.
  • Qwen3-8B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license. Used as the distillation teacher.
  • Converted with llmexport from MNN by Alibaba (Apache-2.0).