Post
3,161
To all the inference providers behind the HuggingFace router - if you want me to use your services you will have to serve native-precision models. I'm not paying for FP8 when I can run in bf16 locally.
Join the conversation
Join the community of Machine Learners and AI enthusiasts.
Sign Up
That’s a fair deployment preference: providers should label the served precision and whether conversion/quantization happens in the router. A native bf16 path is especially useful for checking quality regressions against a local baseline; FP8 can still be valuable for throughput, but it shouldn’t be presented as equivalent by default.
mfw i dont have 192 bajillabytes of hbm69420 ram and would rather take fp8 glm 5.3 over iq4_xs Qwen 3.6 35ba3b
Wonder if they would raise the prices if this is actually added
FP8 vs bf16 is the smaller problem. The bigger one is that the response does not tell you which one you got.
We asked eight open models on hosted endpoints to infer a prompt's structure, then repeated the identical call, caching off. Agreement between repeats ranged from Jaccard 0.39 (gpt-oss:120b) to 0.96 (minimax-m2.1). Only 35 of 127 prompt-model cells reproduced perfectly.
With no served precision or build in the response, a quantization swap and ordinary nondeterminism look the same from the client. And by our audit date 4 of the 8 variants returned HTTP 410, so the run cannot be repeated at all.
Data: https://huggingface.co/datasets/dipankarsarkar/llm-evaluation-self-audit
Would a served-precision field on every router response be enough for you, or is it the bf16 path itself you want?
if I knew the served precision, It would save me the time of comparing the probabilities of a remote qwen 27B against my local native-precision 27B, and likewise for gemma4 31b. I am running qwen locally in native precision for the accuracy. The accuracy is what I'm after. I'd like to expand - using both my local GPU /w native precision 27B AND a remote qwen of the same precision. I don't want reduced precision for this work I'm doing.
It seems to me like they want our eyes to turn into hearts when we see "Qwen 27B" being served, but they aren't telling us they quantized it. I could run quantized qwen at twice the speed of native precision qwen, I don't want quantized.
Your probability check is the right test. It needs two baselines before a gap means anything.
Remote against remote first, the same call twice. Batch size and kernel choice move logprobs even at the same precision, so a provider can look quantized when it is only batched differently. That spread is your noise floor.
Then local against local quantized: your native 27B against your own FP8 of it, same prompts. That is the size of gap a real precision swap leaves.
A remote vs local gap only says "quantized" when it sits outside the first and near the second.
Which are you comparing, top-1 logprobs or the full next-token distribution?