card: add a user's independent image, text and retrieval measurements
Browse files
README.md
CHANGED
|
@@ -151,6 +151,31 @@ raw = torch.tensor(text_embeddings)
|
|
| 151 |
mrl_256 = F.normalize(raw[:, :256], p=2, dim=-1)
|
| 152 |
```
|
| 153 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
## What this build does not change
|
| 155 |
|
| 156 |
It does not touch the int4 tensors, so the dominant quantization error in the model is unchanged. Measured on this model's own shapes, the int4 group-16 grid carries about 9 percent weight error against 0.6 percent for the new int8 grid, which is where the remaining 0.007 gap to the base lives. That gap is documented, not hidden: a wider variant exists only as a measurement, and this repo stays behavior compatible with the FP8 card.
|
|
|
|
| 151 |
mrl_256 = F.normalize(raw[:, :256], p=2, dim=-1)
|
| 152 |
```
|
| 153 |
|
| 154 |
+
## Independent measurements
|
| 155 |
+
|
| 156 |
+
A user ran the base against this build on their own hardware and sent the numbers. Their set is
|
| 157 |
+
289 photos and 244 tag queries, mean cosine of L2-normalized embeddings, same inputs on both
|
| 158 |
+
models, min and max alongside the mean.
|
| 159 |
+
|
| 160 |
+
| Dimension | Image mean | Image min | Text mean | Text min |
|
| 161 |
+
| :--- | ---: | ---: | ---: | ---: |
|
| 162 |
+
| 2048 | 97.2912 | 92.9057 | 99.5509 | 98.6295 |
|
| 163 |
+
| 256 | 98.2565 | 94.8814 | 99.7108 | 98.7139 |
|
| 164 |
+
|
| 165 |
+
256 beats 2048 for them on both axes. Truncating and renormalizing drops the tail dimensions
|
| 166 |
+
where quantization error sits, so the dimension they ship is not the dimension that looks worst.
|
| 167 |
+
|
| 168 |
+
They also measured what the remaining gap is worth in search on the same set, ground truth from a
|
| 169 |
+
tagger unrelated to either model. Text to image mean average precision: base 0.5018, this build
|
| 170 |
+
0.5024, a paired difference of +0.0006 with a 95 percent interval of -0.008 to +0.009. Image to
|
| 171 |
+
image: +0.0024, interval +0.0005 to +0.0043. Going from this build all the way to full precision
|
| 172 |
+
moves mAP by less than 0.01, while keeping 256 dimensions instead of 2048 costs about 0.066.
|
| 173 |
+
|
| 174 |
+
Their 97.29 image figure disagrees with the 94.6120 I published on the
|
| 175 |
+
[FP8 card](https://huggingface.co/ewin-reg/WeMM-Embedding-2B-Quantized). Different input sets,
|
| 176 |
+
and mine is lost, so theirs is the one to use. These are their measurements on their runners and
|
| 177 |
+
they are theirs to correct.
|
| 178 |
+
|
| 179 |
## What this build does not change
|
| 180 |
|
| 181 |
It does not touch the int4 tensors, so the dominant quantization error in the model is unchanged. Measured on this model's own shapes, the int4 group-16 grid carries about 9 percent weight error against 0.6 percent for the new int8 grid, which is where the remaining 0.007 gap to the base lives. That gap is documented, not hidden: a wider variant exists only as a measurement, and this repo stays behavior compatible with the FP8 card.
|