About

This repository provides a model compiled and optimized for Mobilint NPU hardware. The model is packaged for deployment on Mobilint's acceleration stack and is intended to be used within that environment.

multilingual-e5-small-ko-v2-regulus-rb-usb for Mobilint NPUs

dragonkue/multilingual-e5-small-ko-v2 (revision fcfc26bf355882620c48df58be112275bd756f50), a sentence embedding model, compiled for the Mobilint regulus-rb-usb NPU. The embedding lookup runs on the host and the encoder on the NPU; the host then pools the last hidden state.

import torch
import transformers_mblt
from transformers import AutoModel, AutoTokenizer

transformers_mblt.register()
tokenizer = AutoTokenizer.from_pretrained("mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb")
model = AutoModel.from_pretrained("mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb")

texts = ["query: μ—”μ§„ μ˜€μΌμ€ μ–Έμ œ κ΅μ²΄ν•˜λ‚˜μš”?", "passage: μ—”μ§„ μ˜€μΌμ€ 1만 kmλ§ˆλ‹€ κ΅μ²΄ν•©λ‹ˆλ‹€."]
batch = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
hidden = model(**batch).last_hidden_state
mask = batch["attention_mask"].unsqueeze(-1).float()
embeddings = torch.nn.functional.normalize((hidden * mask).sum(1) / mask.sum(1), dim=-1)
print(embeddings[0] @ embeddings[1])
model.dispose()

As the base model, prefix queries with query: and documents with passage: , mean-pool the last hidden state over the attention mask and L2-normalize.

Requirements

  • A Mobilint REGULUS USB board (regulus-rb-usb) with its driver.
  • pip install "transformers-mblt>=0.8.0". It installs mobilint-qb-runtime; this build was tested with qbruntime 1.5.3.
  • The model loads after transformers_mblt.register(), or with trust_remote_code=True (which runs proxy_bert_embedding.py). The tokenizer is the base model's own.
  • Core modes: single only.
  • Sequences of up to 512 tokens. Each sequence of a batch runs on its own, without its padding.

Quantization

Item Setting
Weights 8 bit
Activations 16 bit
Method Post-training quantization calibrated on the texts below
Compiler qbcompiler 1.4.1

Calibration

1,024 texts, 241,255 tokens, at most 512 tokens each, with the base model's query: / passage: prefixes. Shares are of texts. No calibration text appears in the evaluation sets below.

Source Share Licence
KLUE-MRC train split (Korean) 31.3% (questions and contexts) CC BY-SA 4.0
SQuAD train split (English) 28.1% (questions and contexts) CC BY-SA 4.0
FineWeb / FineWeb-2 passages in 11 languages (ar, de, en, es, fr, it, ja, ko, pt, ru, zh) 40.6% ODC-By 1.0

Fidelity

Measured on a REGULUS USB board (regulus-rb-usb, one core) against the base model in fp32 on a CPU, on texts never used for calibration. MTEB scores are the task's main metric; the fp32 score is in brackets.

Set This build
Embedding cosine to fp32, 30,078 texts: mean / lowest 1% / minimum 0.9990 / 0.9983 / 0.985
Cosine to fp32 per language (11 languages) 0.9991–0.9992
Question answering retrieval, KorQuAD/SQuAD validation, 300 queries: R@1 [0.923] / top-1 same as fp32 0.927 / 0.990
KLUE-STS [0.806] / KorSTS [0.781] / STSBenchmark [0.849] 0.807 / 0.781 / 0.848
Ko-StrategyQA [0.768] / NFCorpus [0.316] / SciFact [0.687] 0.767 / 0.317 / 0.682

The same model compiled for aries-rb scores the same within noise (mobilint/multilingual-e5-small-ko-v2).

Time per sequence on regulus-rb-usb: 10.6 ms at 128 tokens, 52.6 ms at 512 tokens. 600 consecutive calls of 16-512 tokens ran without errors, and the same input always gave the same output.

Licence

The base model, dragonkue/multilingual-e5-small-ko-v2, is licensed under the Apache License 2.0; its text is in LICENSE. This repository is a derivative work under the same licence. Changes: the encoder is quantized and compiled into multilingual-e5-small-ko-v2-regulus-rb-usb.mxq; model.safetensors holds the base model's embedding-layer weights only, unchanged; config.json and proxy_bert_embedding.py are new files for transformers-mblt. The tokenizer files (tokenizer.json, tokenizer_config.json, special_tokens_map.json) are the base model's, unchanged.

The base model is fine-tuned from intfloat/multilingual-e5-small, which is released under the MIT License.

Downloads last month
54
Safetensors
Model size
96.2M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for mobilint/multilingual-e5-small-ko-v2-regulus-rb-usb

Quantized
(3)
this model