ยท
AI & ML interests
None yet
Recent Activity
reacted to AC-723's post with ๐ฅ 3 days ago MixtureKit 1.1 is out ๐ Turn the fine-tuned models you already have on the Hub into one Mixture-of-Experts model.
pip install mixturekit
Give it a base model and its fine-tunes (math, code, a language...). MixtureKit merges them, trains the routers that pick an expert for every token, and shows you what they learned:
๐งฉ 3 composition methods behind one YAML: Branch-Train-MiX, Mixtral-style MoE, Branch-Train-Stitch
๐ป Merges stream the experts tensor by tensor: they run on a laptop
๐ฏ Router training with a load balancing loss, optional LoRA
๐ A routing visualizer: in the image, two Nile 4B experts split a mixed Egyptian sentence, Arabic script to one expert, Arabizi to the other
๐ฆ Export to Mixtral / Qwen3-MoE, then GGUF for llama.cpp
๐ค mixturekit push uploads the result with a generated model card
โ
Llama 3, Qwen 2.5/3/3.5, Gemma 3/4 experts; tested on transformers 4.52 to 5.19
In the paper, a BTX model built with MixtureKit outperforms the dense baselines on multiple benchmarks.
โถ๏ธ Build, train and visualize an MoE in 15 min on a free Colab GPU: https://colab.research.google.com/github/MBZUAI-Paris/MixtureKit/blob/main/examples/build_your_own_moe.ipynb
๐ Paper: https://huggingface.co/papers/2512.12121
๐ป Code: https://github.com/MBZUAI-Paris/MixtureKit
๐ Docs: https://mbzuai-paris.github.io/MixtureKit/
Work with @guokan-shang and @oe2015 at @MBZUAI-Paris ๐ View all activity Organizations
salma-remyx/spaceom_qwen3_2B
Updated
salma-remyx/spaceom-qwen3-vl-2b-merged
Image-Text-to-Text
โข 2B โข Updated โข 16
salma-remyx/spaceom-qwen3-vl-4b-merged
Image-Text-to-Text
โข 4B โข Updated โข 28
salma-remyx/spaceom_qwen3_4B
Updated
salma-remyx/spacethinker-qwen3-4B-lora
Updated
salma-remyx/DeepSeek-R1-Distill-Qwen-1.5B-GRPO
Updated
salma-remyx/MindCube_train_plain_cgmap_out_qwen_sft.json
Updated
salma-remyx/MindCube_train_plain_cgmap_ffr_out_qwen_sft.json
Updated
salma-remyx/MindCube_train_ff_rsn_qwen_sft.json
Updated
salma-remyx/MindCube_train_cgmap_in_ffr_out_qwen_sft.json
Updated
salma-remyx/MindCube_train_aug_cgmap_out_qwen_sft.json
Updated
salma-remyx/MindCube_train_aug_cgmap_in_qwen_sft.json
Updated
salma-remyx/MindCube_train_aug_cgmap_ffr_out_qwen_sft.json
Updated
salma-remyx/MindCube_tinybench_raw_qa_qwen_sft.json
Updated
salma-remyx/MindCube_tinybench_plain_cgmap_out_qwen_sft.json
Updated
salma-remyx/MindCube_tinybench_plain_cgmap_ffr_out_qwen_sft.json
Updated
salma-remyx/MindCube_tinybench_ff_rsn_qwen_sft.json
Updated
salma-remyx/MindCube_tinybench_cgmap_in_ffr_out_qwen_sft.json
Updated
salma-remyx/MindCube_tinybench_aug_cgmap_out_qwen_sft.json
Updated
salma-remyx/MindCube_tinybench_aug_cgmap_ffr_out_qwen_sft.json
Updated
salma-remyx/MindCube_tinybench_aug_cgmap_in_qwen_sft.json
Updated
salma-remyx/SpaceOm_results
Updated
salma-remyx/spacellava-1.5-7b
Image-Text-to-Text
โข 7B โข Updated โข 12
โข 1
salma-remyx/SpaceThinker-Qwen2.5VL-7B
8B โข Updated โข 13
salma-remyx/spacethinker-qwen2.5-3b
Updated โข 6
Feature Extraction
โข 0.2B โข Updated โข 6
โข 1
salma-remyx/test_train_general_1
Updated โข 8
Image-Text-to-Text
โข 0.3B โข Updated โข 9
โข 1
salma-remyx/SpaceQwen2-VL-7B-Instruct
Image-Text-to-Text
โข 9B โข Updated โข 10
salma-remyx/spaceqwen2-7b-instruct
Updated