Feature Extraction
sentence-transformers
Safetensors
English
Chinese
multilingual
qwen3_5
multimodal
embeddings
retrieval
quantization
mixed-precision
int8
w8a8
mrl
cross-modal
text-embeddings
image-embedding
video-embedding
custom_code
Eval Results (legacy)
8-bit precision
Instructions to use ewin-reg/WeMM-Embedding-2B-INT8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use ewin-reg/WeMM-Embedding-2B-INT8 with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("ewin-reg/WeMM-Embedding-2B-INT8", trust_remote_code=True) sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Notebooks
- Google Colab
- Kaggle
Card fixes: 900 tensors, lm_head section, measured scale accounting
Browse files
README.md
CHANGED
|
@@ -44,7 +44,7 @@ model-index:
|
|
| 44 |
|
| 45 |
# WeMM-Embedding-2B-INT8
|
| 46 |
|
| 47 |
-
INT8 rebuild of [ewin-reg/WeMM-Embedding-2B-Quantized](https://huggingface.co/ewin-reg/WeMM-Embedding-2B-Quantized). Same model,
|
| 48 |
|
| 49 |
## Why this exists
|
| 50 |
|
|
@@ -65,7 +65,9 @@ Variant of [tencent/WeMM-Embedding-2B](https://huggingface.co/tencent/WeMM-Embed
|
|
| 65 |
| `mlp.{gate_proj,up_proj}` | 48 | uint8 packed int4, group 16 | 377.9 MB |
|
| 66 |
| `visual.blocks` and `merger` | 98 | uint8 packed int4, group 16 | 219.8 MB |
|
| 67 |
| norms, conv1d, scales | misc | BF16 and F32 | 18.5 MB |
|
| 68 |
-
| **Checkpoint total** | **
|
|
|
|
|
|
|
| 69 |
|
| 70 |
The int4 tensors are copied byte for byte from the FP8 checkpoint. Unpacking reads even columns from the low nibble and odd columns from the high nibble, which is the order that checkpoint ships.
|
| 71 |
|
|
@@ -82,6 +84,10 @@ Three sources back this exact choice. SLQ, arXiv 2605.02404, proves symmetric gr
|
|
| 82 |
|
| 83 |
CUDA, MPS and CPU all run from the same files with stock `torch`. The FP8 forward dispatched `F.embedding` on a float8 weight, which is the call with no MPS kernel; the INT8 embedding reads rows with `index_select`, which takes integer dtypes on all three backends. Matrix weights dequantize to the input dtype inside each forward call, weight only, activations stay in full width throughout. Nothing in this repo imports the fp8 MPS shim package.
|
| 84 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
## Measured fidelity
|
| 86 |
|
| 87 |
24 text queries through the unquantized base, the FP8 checkpoint and this build, one shared protocol, CPU runtime, no prompt template, embeddings read from each model's own `embedding` method and compared as cosines:
|
|
|
|
| 44 |
|
| 45 |
# WeMM-Embedding-2B-INT8
|
| 46 |
|
| 47 |
+
INT8 rebuild of [ewin-reg/WeMM-Embedding-2B-Quantized](https://huggingface.co/ewin-reg/WeMM-Embedding-2B-Quantized). Same model, 139 FP8 tensors re-encoded as asymmetric int8. 1.94 GB on disk, runs under plain PyTorch on CPU, CUDA and MPS with no kernel shim.
|
| 48 |
|
| 49 |
## Why this exists
|
| 50 |
|
|
|
|
| 65 |
| `mlp.{gate_proj,up_proj}` | 48 | uint8 packed int4, group 16 | 377.9 MB |
|
| 66 |
| `visual.blocks` and `merger` | 98 | uint8 packed int4, group 16 | 219.8 MB |
|
| 67 |
| norms, conv1d, scales | misc | BF16 and F32 | 18.5 MB |
|
| 68 |
+
| **Checkpoint total** | **900 tensors** | mixed custom | **1.94 GB** |
|
| 69 |
+
|
| 70 |
+
The file carries 900 tensors. The two missing against the source are F32 housekeeping buffers, 0.2 MB combined, dropped with the old FP8-era scale tensors. Fidelity was measured on the file as shipped.
|
| 71 |
|
| 72 |
The int4 tensors are copied byte for byte from the FP8 checkpoint. Unpacking reads even columns from the low nibble and odd columns from the high nibble, which is the order that checkpoint ships.
|
| 73 |
|
|
|
|
| 84 |
|
| 85 |
CUDA, MPS and CPU all run from the same files with stock `torch`. The FP8 forward dispatched `F.embedding` on a float8 weight, which is the call with no MPS kernel; the INT8 embedding reads rows with `index_select`, which takes integer dtypes on all three backends. Matrix weights dequantize to the input dtype inside each forward call, weight only, activations stay in full width throughout. Nothing in this repo imports the fp8 MPS shim package.
|
| 86 |
|
| 87 |
+
### lm_head.weight is absent by design
|
| 88 |
+
|
| 89 |
+
`config.json` sets `tie_word_embeddings: true`, so `lm_head.weight` is not stored and ties to `embed_tokens` at load. Under `device_map="auto"` the meta placeholder never materializes, which is why a device dump shows `lm_head` on the meta device. Nothing is missing and the forward pass is unaffected.
|
| 90 |
+
|
| 91 |
## Measured fidelity
|
| 92 |
|
| 93 |
24 text queries through the unquantized base, the FP8 checkpoint and this build, one shared protocol, CPU runtime, no prompt template, embeddings read from each model's own `embedding` method and compared as cosines:
|