ewin-reg commited on
Commit
bde0d4c
·
verified ·
1 Parent(s): 9d18deb

Card fixes: 900 tensors, lm_head section, measured scale accounting

Browse files
Files changed (1) hide show
  1. README.md +8 -2
README.md CHANGED
@@ -44,7 +44,7 @@ model-index:
44
 
45
  # WeMM-Embedding-2B-INT8
46
 
47
- INT8 rebuild of [ewin-reg/WeMM-Embedding-2B-Quantized](https://huggingface.co/ewin-reg/WeMM-Embedding-2B-Quantized). Same model, same mixed allocation, eight mantissa bits everywhere the FP8 build used three. 1.94 GB on disk, runs under plain PyTorch on CPU, CUDA and MPS with no kernel shim.
48
 
49
  ## Why this exists
50
 
@@ -65,7 +65,9 @@ Variant of [tencent/WeMM-Embedding-2B](https://huggingface.co/tencent/WeMM-Embed
65
  | `mlp.{gate_proj,up_proj}` | 48 | uint8 packed int4, group 16 | 377.9 MB |
66
  | `visual.blocks` and `merger` | 98 | uint8 packed int4, group 16 | 219.8 MB |
67
  | norms, conv1d, scales | misc | BF16 and F32 | 18.5 MB |
68
- | **Checkpoint total** | **902 tensors** | mixed custom | **1.94 GB** |
 
 
69
 
70
  The int4 tensors are copied byte for byte from the FP8 checkpoint. Unpacking reads even columns from the low nibble and odd columns from the high nibble, which is the order that checkpoint ships.
71
 
@@ -82,6 +84,10 @@ Three sources back this exact choice. SLQ, arXiv 2605.02404, proves symmetric gr
82
 
83
  CUDA, MPS and CPU all run from the same files with stock `torch`. The FP8 forward dispatched `F.embedding` on a float8 weight, which is the call with no MPS kernel; the INT8 embedding reads rows with `index_select`, which takes integer dtypes on all three backends. Matrix weights dequantize to the input dtype inside each forward call, weight only, activations stay in full width throughout. Nothing in this repo imports the fp8 MPS shim package.
84
 
 
 
 
 
85
  ## Measured fidelity
86
 
87
  24 text queries through the unquantized base, the FP8 checkpoint and this build, one shared protocol, CPU runtime, no prompt template, embeddings read from each model's own `embedding` method and compared as cosines:
 
44
 
45
  # WeMM-Embedding-2B-INT8
46
 
47
+ INT8 rebuild of [ewin-reg/WeMM-Embedding-2B-Quantized](https://huggingface.co/ewin-reg/WeMM-Embedding-2B-Quantized). Same model, 139 FP8 tensors re-encoded as asymmetric int8. 1.94 GB on disk, runs under plain PyTorch on CPU, CUDA and MPS with no kernel shim.
48
 
49
  ## Why this exists
50
 
 
65
  | `mlp.{gate_proj,up_proj}` | 48 | uint8 packed int4, group 16 | 377.9 MB |
66
  | `visual.blocks` and `merger` | 98 | uint8 packed int4, group 16 | 219.8 MB |
67
  | norms, conv1d, scales | misc | BF16 and F32 | 18.5 MB |
68
+ | **Checkpoint total** | **900 tensors** | mixed custom | **1.94 GB** |
69
+
70
+ The file carries 900 tensors. The two missing against the source are F32 housekeeping buffers, 0.2 MB combined, dropped with the old FP8-era scale tensors. Fidelity was measured on the file as shipped.
71
 
72
  The int4 tensors are copied byte for byte from the FP8 checkpoint. Unpacking reads even columns from the low nibble and odd columns from the high nibble, which is the order that checkpoint ships.
73
 
 
84
 
85
  CUDA, MPS and CPU all run from the same files with stock `torch`. The FP8 forward dispatched `F.embedding` on a float8 weight, which is the call with no MPS kernel; the INT8 embedding reads rows with `index_select`, which takes integer dtypes on all three backends. Matrix weights dequantize to the input dtype inside each forward call, weight only, activations stay in full width throughout. Nothing in this repo imports the fp8 MPS shim package.
86
 
87
+ ### lm_head.weight is absent by design
88
+
89
+ `config.json` sets `tie_word_embeddings: true`, so `lm_head.weight` is not stored and ties to `embed_tokens` at load. Under `device_map="auto"` the meta placeholder never materializes, which is why a device dump shows `lm_head` on the meta device. Nothing is missing and the forward pass is unaffected.
90
+
91
  ## Measured fidelity
92
 
93
  24 text queries through the unquantized base, the FP8 checkpoint and this build, one shared protocol, CPU runtime, no prompt template, embeddings read from each model's own `embedding` method and compared as cosines: