Instructions to use mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb", trust_remote_code=True, device_map="auto") - Mobilint
How to use mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb with Mobilint:
# pip install mblt-model-zoo from mblt_model_zoo.vision import MBLT_Engine model = MBLT_Engine( model_cls="multilingual-e5-small-ko-v2-regulus-rb-usb", model_type="DEFAULT", model_path="", core_mode="global8", ) try: image = model.preprocess("path/to/image.jpg") output = model(image) result = model.postprocess(output) finally: model.dispose() - Notebooks
- Google Colab
- Kaggle
About
This repository provides a model compiled and optimized for Mobilint NPU hardware. The model is packaged for deployment on Mobilint's acceleration stack and is intended to be used within that environment.
multilingual-e5-small-ko-v2-regulus-rb-usb for Mobilint NPUs
dragonkue/multilingual-e5-small-ko-v2 (revision fcfc26bf355882620c48df58be112275bd756f50), a sentence embedding model, compiled for the Mobilint
regulus-rb-usb NPU. The embedding lookup runs on the host and the encoder on the NPU; the host then pools the last hidden state.
import torch
import transformers_mblt
from transformers import AutoModel, AutoTokenizer
transformers_mblt.register()
tokenizer = AutoTokenizer.from_pretrained("mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb")
model = AutoModel.from_pretrained("mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb")
texts = ["query: μμ§ μ€μΌμ μΈμ κ΅μ²΄νλμ?", "passage: μμ§ μ€μΌμ 1λ§ kmλ§λ€ κ΅μ²΄ν©λλ€."]
batch = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
hidden = model(**batch).last_hidden_state
mask = batch["attention_mask"].unsqueeze(-1).float()
embeddings = torch.nn.functional.normalize((hidden * mask).sum(1) / mask.sum(1), dim=-1)
print(embeddings[0] @ embeddings[1])
model.dispose()
As the base model, prefix queries with query: and documents with passage: , mean-pool the last hidden state over
the attention mask and L2-normalize.
Requirements
- A Mobilint REGULUS USB board (
regulus-rb-usb) with its driver. pip install "transformers-mblt>=0.8.0". It installsmobilint-qb-runtime; this build was tested with qbruntime 1.5.3.- The model loads after
transformers_mblt.register(), or withtrust_remote_code=True(which runsproxy_bert_embedding.py). The tokenizer is the base model's own. - Core modes:
singleonly. - Sequences of up to 512 tokens. Each sequence of a batch runs on its own, without its padding.
Quantization
| Item | Setting |
|---|---|
| Weights | 8 bit |
| Activations | 16 bit |
| Method | Post-training quantization calibrated on the texts below |
| Compiler | qbcompiler 1.4.1 |
Calibration
1,024 texts, 241,255 tokens, at most 512 tokens each, with the base model's query: / passage: prefixes. Shares are of texts. No calibration text appears in the evaluation sets below.
| Source | Share | Licence |
|---|---|---|
| KLUE-MRC train split (Korean) | 31.3% (questions and contexts) | CC BY-SA 4.0 |
| SQuAD train split (English) | 28.1% (questions and contexts) | CC BY-SA 4.0 |
| FineWeb / FineWeb-2 passages in 11 languages (ar, de, en, es, fr, it, ja, ko, pt, ru, zh) | 40.6% | ODC-By 1.0 |
Fidelity
Measured on a REGULUS USB board (regulus-rb-usb, one core) against the base model in fp32 on a CPU, on texts never used for calibration. MTEB scores are the task's main metric; the fp32 score is in brackets.
| Set | This build |
|---|---|
| Embedding cosine to fp32, 30,078 texts: mean / lowest 1% / minimum | 0.9990 / 0.9983 / 0.985 |
| Cosine to fp32 per language (11 languages) | 0.9991β0.9992 |
| Question answering retrieval, KorQuAD/SQuAD validation, 300 queries: R@1 [0.923] / top-1 same as fp32 | 0.927 / 0.990 |
| KLUE-STS [0.806] / KorSTS [0.781] / STSBenchmark [0.849] | 0.807 / 0.781 / 0.848 |
| Ko-StrategyQA [0.768] / NFCorpus [0.316] / SciFact [0.687] | 0.767 / 0.317 / 0.682 |
The same model compiled for aries-rb scores the same within noise (mobilint/multilingual-e5-small-ko-v2).
Time per sequence on regulus-rb-usb: 10.6 ms at 128 tokens, 52.6 ms at 512 tokens. 600 consecutive calls of 16-512 tokens ran without errors, and the same input always gave the same output.
Licence
The base model, dragonkue/multilingual-e5-small-ko-v2, is licensed under the Apache License 2.0; its
text is in LICENSE. This repository is a derivative work under the same licence. Changes: the encoder is
quantized and compiled into multilingual-e5-small-ko-v2-regulus-rb-usb.mxq; model.safetensors holds the base model's embedding-layer weights only,
unchanged; config.json and proxy_bert_embedding.py are new files for transformers-mblt.
The tokenizer files (tokenizer.json, tokenizer_config.json, special_tokens_map.json) are the base model's, unchanged.
The base model is fine-tuned from intfloat/multilingual-e5-small, which is released under the MIT License.
- Downloads last month
- 54
Model tree for mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb
Base model
intfloat/multilingual-e5-small