SozKZ MoE: Mixture of Experts
Collection
Mixture-of-Experts models for Kazakh — upcycled and domain-pretrained MoE architectures • 4 items • Updated
kazakh-moe-160M-A50M-domain
A Kazakh language model trained from scratch on a curated, deduplicated Kazakh text corpus (~1B tokens). Llama architecture with Chinchilla-optimal compute budget.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("stukenov/kazakh-moe-160M-A50M-domain")
model = AutoModelForCausalLM.from_pretrained("stukenov/kazakh-moe-160M-A50M-domain")
text = "Қазақстан — "
inputs = tokenizer(text, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))
Architecture
| Parameter | Value |
|---|---|
| Type | mixtral |
| Parameters | ~51M |
| Hidden size | 512 |
| Layers | 8 |
| Attention heads | 8 |
| Vocab size | 50,257 |
| Context length | 1024 |
Training Data
Trained on stukenov/kazakh-clean-pretrain-v2 — a curated Kazakh corpus processed through a 9-stage cleaning pipeline:
- NFC normalization + whitespace collapsing
- Kazakh character verification
- Script profile filtering (Cyrillic >= 60%)
- fastText language identification (kk >= 0.5)
- Junk removal (URLs, HTML, boilerplate)
- Repetition filtering
- Exact + near deduplication (MinHash LSH)
- Domain balancing
Sources: CC-100, OSCAR, Wikipedia, Leipzig, Kazakh News, Kazakh Books.
Training
See the SLM project for full experiment details.
- Downloads last month
- -
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support