# sam3-p150 — Blackhole p150a vs RTX 5090 (same host, same weights, same input)

Date 2026-09-14. Facts only; every GPU number below was measured in this pass, every p150a number is copied (with its
source line) from the validation / publish reports. The p150a was NOT touched.

## What was run

| | |
|---|---|
| Model | Meta SAM3 image model (`sam3` 0.1.4 `Sam3Image`: ViT-H/14 backbone at 1008 (32 blocks, RoPE, windowed + 4 global attention) + FPN neck + CLIP-style text encoder + 6-layer fusion encoder + DETR decoder (200 queries) + segmentation head; 840.51 M parameters), built exactly as the port's server builds its host reference: `tt_sam3.tt.ttnn_sam3_model.build_sam3_model(use_pretrained=True, checkpoint_path=<sam3.pt>)` = `sam3.model_builder.build_sam3_image_model(device="cpu", eval_mode=True, enable_segmentation=True, enable_inst_interactivity=False)`, then `.to("cuda")`. This is the "torch CPU fp32 reference (pristine model)" the p150a port was gated against (`models/sam3-p150/DEVICE_VALIDATION.md` "## Results"). The forward is `Sam3Image.forward(make_batched_datapoint(pixel_values, ["car"]))` -> `extract_predictions` — the `pipeline.forward_image(px, [prompt])` path `server/app.py::predict` times as `timing_ms.forward`. |
| Weights | `facebook/sam3` @ `3c879f39826c281e95690f02c7821c4de09afae7` (`tt-model.yaml` `weights.revision`, asserted by the script), `sam3.pt` 3.45 GB from `~/.cache/huggingface/hub/models--facebook--sam3/snapshots/3c879f39…/sam3.pt`, `HF_HUB_OFFLINE=1` (cache hit, nothing downloaded); BPE vocab = the port's vendored `tt_sam3/assets/bpe_simple_vocab_16e6.txt.gz`. |
| Input | `media/source_1.png` (1920x1080 RGBA street scene; md5 identical to `logs/megakernel-validate/sam3/data/sam3_demo.png`, the image of the validation probes) -> `server/app.py::_to_pixel_values` copied verbatim (PIL RGB, /255, bilinear squash `align_corners=False` to 1008x1008, `(x-0.5)/0.5`) -> `pixel_values [1,3,1008,1008]` fp32 (12.2 MB); text prompt `"car"` (the demo prompt); batch 1, one prompt; server defaults threshold 0.5, `max_instances` 50, `mask_format` png. |
| GPU | NVIDIA GeForce RTX 5090 (sm_120), driver 580.126.18, power limit 600 W, 32607 MiB; idle 30.5 W. |
| venv | `/home/deepgadget/experiments/tt-models/.venv-gpu/main` — Python 3.12.13, torch 2.11.0+cu128, CUDA 12.8, cuDNN 9.19.0, torchvision 0.26.0, sam3 0.1.4, timm 1.0.29, triton 3.6.0, numpy 1.26.4, pillow 12.3.0, einops 0.8.2, iopath 0.1.10, huggingface_hub 1.31.0. |
| Host CPU (CPU reference / host stages) | AMD Ryzen 9 7950X 16-core, torch 16 intra-op threads. |
| Script / logs | `logs/gpu-vs-p150/sam3/bench_sam3_gpu.py` (uses `logs/gpu-vs-p150/bench_common.py`); `full_run.log` (main run), `compile_retry.log` (`--only-compile` retry of one compile variant, merged into the same JSON), `smoke.log` / `shakedown.log` (development runs); raw JSON `reports/gpu-vs-p150/sam3.json` (= `logs/gpu-vs-p150/sam3/result.json`; `result_main_run.json` = before the compile retry); CPU reference tensors `logs/gpu-vs-p150/sam3/cpu_fp32_reference.pt`. |
| Commands | `HF_HUB_OFFLINE=1 .venv-gpu/main/bin/python bench_sam3_gpu.py --iters 50 --warmup 10 --served-iters 50`, then `… --iters 50 --warmup 10 --only-compile`. |
| Loop | per precision and per stage: 10 warm-ups + 50 timed iterations, `torch.cuda.synchronize()` before/after each; wall-clock (`perf_counter`) is the primary number, CUDA-event time recorded alongside (within 0.05 ms of wall on every excl loop); pageable host tensors (what the server's preprocess produces). |
| p150a source | `reports/gpu-vs-p150/p150_numbers.json` -> `reports/megakernel/PUBLISH_SUMMARY.md:19` (Hub `tt serve`: forward 486.8 ms, total 541.5 ms), `reports/megakernel/VALIDATION_SUMMARY.md:36` (verifier re-run: forward 476.8 / 463.2 / 490.7, total 532.0), `models/sam3-p150/DEVICE_VALIDATION.md` "## Results" (EF default row: backbone bench 89.5 / 88.6 ms, island probe backbone 90.0 ms + encoder 16.8 ms, probe forward median 413-459 ms, served 471.6-489.8 ms, demo IoU288 0.9989, top 0.9729, mask PCC vs ref 0.984), `models/sam3-p150/README.md` "### Performance" (ViT backbone 89 ms / 11.2 FPS, fusion encoder 16.8 ms, served forward ~480 ms (472-490); CPU fp32 reference ~6.0 s on this host). |

The p150a's published pipeline is a SPLIT pipeline: only the ViT-H backbone and the 6-layer fusion encoder run on the chip
(bf16 activations, bf8/bf16 weights, one metal trace each); the FPN neck, text encoder (cached per prompt), DETR decoder and
segmentation head run on the host CPU in fp32 (`/info.hardware` in `server/app.py`; `p150_numbers.json`: "~65 % of the request is
CPU"). So `timing_ms.forward` = 486.8 ms is ~106 ms of device work + ~380 ms of host CPU work. The GPU runs the whole network
on the device; the stage split below gives the like-for-like device numbers.

Timing definitions (matching the p150a `timing_ms` keys and the DEVICE_VALIDATION island probes):

- **full incl_h2d (text cached)** = `pixel_values.to("cuda")` + `Sam3Image.forward` + readback of `pred_masks [1,200,288,288]`, `pred_logits`, `pred_boxes`, `presence_logit_dec` (66.4 MB fp32; 33.2 MB when autocast leaves the masks in bf16/fp16). The text features for the prompt are cached after the first call, exactly as `TtSam3ImagePipeline._cached_forward_text` does in the served pipeline. Compare with p150a `timing_ms.forward` (**486.8 ms**, PUBLISH_SUMMARY:19) = `pipeline.forward_image(px, [prompt])` incl. upload, device backbone + encoder, bf16 neck hand-off readback and the CPU neck / decoder / seg head.
- **full excl_h2d** = forward only, input resident, outputs left on the device.
- **full, text uncached** = the same with the CLIP text encoder run on every call (not what the server does after the first request; informational).
- **backbone incl_h2d** = `pixel_values.to("cuda")` + ViT trunk (`patch_embed` + 32 blocks + `ln_post`) + readback of the `[1,1024,72,72]` fp32 features (21.2 MB). Compare with the p150a **backbone bench 89.5 ms** (DEVICE_VALIDATION EF row: `benchmark.py --mode backbone`, `tt_vit_backbone_fused(pixel_values, params, device)` host in / host out; island probe 90.0 ms; README 89 ms). The GPU reads back 21 MB fp32, the p150a hands off bf16.
- **backbone excl_h2d** = the trunk only, resident I/O.
- **encoder incl / excl** = `TransformerEncoderFusion.forward` on the inputs captured from the real forward (`src`/`src_pos` `[5184,1,256]`, prompt `[33,1,256]`, 10.7 MB in / 10.7 MB out), incl. = host copies uploaded + outputs read back. Compare with the p150a **encoder island 16.8 ms** (DEVICE_VALIDATION EF row, `probe_islands.py`: host tensors in, `TtFusedEncoder` trace, host out).
- **served-like** = base64 decode + PNG decode (PIL) + `_to_pixel_values` + full incl_h2d forward + `select_instances` (0.5, 50) + `_postprocess_prompt(mask_format="png")` (bilinear upsample of each kept 288x288 mask logit to 1920x1080, `> 0`, PNG encode, base64) — the host work of `server/app.py::predict`, copied verbatim (fastapi is not in the venv). Compare with p150a `timing_ms.total` (**541.5 ms**, PUBLISH_SUMMARY:19).

## Correctness check (GPU vs CPU fp32 reference)

CPU fp32 forward on this host: 7274 / 7430 ms (two calls, bit-identical; 16 threads — the README card says ~6.0 s at 17 threads;
the 6.0 s smoke call in `smoke.log` agrees). Top score 0.9716 (`sigmoid(logit) * sigmoid(presence)`), 3 instances > 0.5,
presence logit 5.3437. The freshly computed reference matches the port's cached validation reference
(`logs/megakernel-validate/sam3/ref/sam3_demo__car.pt`, computed in the shipped container): PCC masks 0.999999999, logits
0.999999998, boxes 1.0, top-mask IoU288 1.0 — same model, same weights, same preprocessing.

GPU fp32 strict (no TF32, text encoder run on the GPU) on the same `pixel_values`:

| metric | value |
|---|---:|
| PCC `pred_masks` `[200,288,288]` / `pred_logits` `[200]` / `pred_boxes` `[200,4]` | **1.000000 / 1.000000 / 1.000000** |
| max abs diff mask logits / class logits / boxes | 0.121 / 0.0021 / 1.1e-4 |
| presence logit GPU / CPU | 5.343726 / 5.343727 |
| top score GPU / CPU, instances > 0.5 | 0.9716 / 0.9716, 3 / 3 |
| top-mask IoU (288x288 and at 1920x1080) | 1.0 / 1.0 |

PCC > 0.999 holds for fp32; the GPU runs the right model. Two implementation details were needed to move the CPU-built model to
CUDA (no upstream code changed): `Sam3Image.device` caches `_device` and `.cuda()` bypasses its `.to()` override, and two
non-buffer tensor caches (`PositionEmbeddingSine.cache`, precomputed at build under the port's cuda->cpu patch, and the decoder's
`compilable_cord_cache`, created lazily during the CPU call) stay on the CPU — the script resets / moves them
(`move_plain_attrs`) and builds the datapoint's empty geometry-prompt tensors on the GPU.

Per-precision accuracy vs the CPU fp32 reference (whole network under that precision, text encoder included):

| GPU precision | PCC masks | PCC logits | PCC boxes | max abs diff masks | top score (ref 0.9716) | inst > 0.5 (ref 3) | top-mask IoU288 | PCC > 0.999 gate | port task gates |
|---|---:|---:|---:|---:|---:|---:|---:|---|---|
| fp32 strict | 1.000000 | 1.000000 | 1.000000 | 0.12 | 0.9716 | 3 | 1.0 | yes | yes |
| tf32 | 0.999775 | 0.999684 | 0.999984 | 24.7 | 0.9716 | 3 | 1.0 | yes | yes |
| bf16 autocast | 0.987695 | 0.945449 | 0.990816 | 131.8 | 0.9712 | 3 | 0.99972 | **no** | yes |
| fp16 autocast | 0.997768 | 0.997640 | 0.999582 | 72.4 | 0.9716 | 3 | 1.0 | **no** (0.9978) | yes |
| bf16 autocast + compiled trunk (default / reduce-overhead) | 0.986095 | 0.941186 | | | 0.9717 | 3 | 0.99986 | no | yes |
| bf16 autocast + compiled trunk+encoder+decoder+pixel-decoder | 0.985181 | 0.939134 | 0.990587 | 139.2 | 0.9715 | 3 | 0.99986 | no | yes |
| p150a fused default (DEVICE_VALIDATION EF row; bf16 chip backbone + encoder, fp32 CPU rest) | 0.984 (mask PCC vs ref, README) | | | | 0.9729 | 3 | 0.9989 | n/a | yes (its gate) |

The two autocast modes miss the brief's PCC > 0.999 gate on the 288x288 mask logits and, above all, on the 200 DETR class logits
(bf16: 0.945) — the low-scoring queries move a lot, the kept instances hardly at all (top score within 0.0004, same 3
instances, top-mask IoU 0.9997-1.0, boxes within 0.6 px at 1920x1080 in the served-like run). The p150a port itself documents that
whole-output PCC is the wrong yardstick for this model ("the legacy path itself sits at mask PCC 0.927-0.972", DEVICE_VALIDATION
"Gate notes") and gated on the task metrics (top-mask IoU288, top score, instance count); by that yardstick every GPU precision passes
and bf16 autocast (IoU288 0.99972, PCC 0.988) sits slightly ABOVE the p150a's fused default (IoU288 0.9989, mask PCC 0.984). fp16 autocast
is finite everywhere and closer to fp32 than bf16 (PCC 0.9978 / 0.9976). Both are timed and labelled as "PCC gate not met".

## GPU latency (batch 1, 1008x1008, one prompt; median / min / p90 of 50 iterations, wall-clock ms)

Eager PyTorch, full network, text features cached per prompt (the served definition):

| precision | full incl_h2d median / min / p90 | full excl_h2d median / min / p90 | CUDA-event excl | first call ms | power W (excl loop) | power W (incl loop) | GPU util % (excl) | peak mem alloc / reserved MiB | readback bytes |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| fp32 strict (`allow_tf32=False`, `'highest'`) | **148.753** / 146.781 / 156.492 | **143.118** / 141.773 / 149.017 | 143.10 | 147.0 (320.2 on the very first call incl. CUDA/cuDNN init) | 589.2 | 570.6 | 96 | 4334 / 4554 | 66.4 MB fp32 |
| tf32 (`allow_tf32=True`, `'high'`; PyTorch default is `'highest'`) | **110.832** / 106.639 / 114.560 | **105.518** / 101.591 / 106.370 | 105.45 | 109.5 | 501.4 | 477.5 | 95 | 4339 / 4554 | 66.4 MB fp32 |
| bf16 autocast (+TF32 remainder) — PCC gate not met | **55.311** / 55.132 / 55.797 | **52.487** / 52.342 / 53.073 | 52.47 | 59.0 | 469.9 | 458.4 | 87 | 4202 / 4554 | 33.2 MB (masks/logits bf16) |
| fp16 autocast (+TF32 remainder) — PCC gate not met | **55.721** / 54.833 / 57.101 | **52.195** / 52.018 / 52.783 | 52.18 | 60.2 | 497.0 | 466.6 | 87 | 4202 / 4554 | 33.2 MB (masks/logits fp16) |

Same loops with the text encoder run on every call (informational; the server caches it):

| precision | full incl_h2d | full excl_h2d | delta vs cached (incl) |
|---|---:|---:|---:|
| fp32 strict | 152.543 / 150.966 / 159.102 | 147.429 / 146.895 / 154.108 | +3.8 ms |
| tf32 | 115.125 / 111.089 / 118.840 | 106.370 / 105.687 / 109.207 | +4.3 ms |
| bf16 autocast | 61.142 / 60.775 / 62.687 | 58.140 / 57.952 / 59.489 | +5.8 ms |
| fp16 autocast | 60.826 / 60.570 / 61.833 | 57.984 / 57.840 / 58.537 | +5.1 ms |

Stage split at the p150a device islands (same loops; "incl" = stage input `.to(cuda)` + stage + stage output `.cpu()`, "excl" = stage only, resident I/O):

| precision | backbone (ViT trunk) incl median / min / p90 | backbone excl | backbone power W (excl) / util % | fusion encoder incl | fusion encoder excl | backbone + encoder incl |
|---|---:|---:|---:|---:|---:|---:|
| fp32 strict | **110.595** / 109.510 / 116.402 | 111.988 / 107.936 / 112.738 | 600.3 (at the 600 W limit) / 99 | **12.452** / 12.279 / 13.689 | 10.888 / 10.774 / 12.300 | 123.0 |
| tf32 | **75.667** / 75.154 / 76.732 | 73.335 / 73.217 / 74.016 | 570.1 / 96 | **11.053** / 10.939 / 11.128 | 9.566 / 9.545 / 9.575 | 86.7 |
| bf16 autocast | **35.681** / 35.502 / 35.875 | 33.833 / 33.788 / 33.921 | 581.7 / 98 | **4.047** / 4.000 / 4.136 | 2.641 / 2.624 / 2.653 | 39.7 |
| fp16 autocast | **35.101** / 34.846 / 35.543 | 33.665 / 33.293 / 33.765 | 601.5 / 96 | **4.304** / 4.258 / 4.329 | 2.892 / 2.864 / 2.946 | 39.4 |
| p150a fused default (DEVICE_VALIDATION EF row / README) | **89.5** (bench; island probe 90.0; legacy 143.1) | | not measured | **16.8** (island probe; legacy 28.3) | | 106.3 |

(The fp32-strict backbone excl median is 1.4 ms above its incl median: both loops sit at the 600 W power limit and the clock
throttles; the min values order as expected, 107.9 vs 109.5.) The remainder of the GPU full forward outside the two islands
(neck + cached text features + decoder + segmentation head + Python dispatch) is 143.1 − (112.0 + 10.9) ≈ 20 ms in fp32 strict
and 52.5 − (33.8 + 2.6) ≈ 16 ms in bf16 — the part that takes ~380 ms on the p150a host CPU.

`torch.compile` (inductor, bf16 autocast — chosen as the fastest eager autocast mode passing the port's task gates, since neither
autocast mode meets PCC > 0.999; every compile finished in 12-18 s, well inside the 5-min budget):

| variant | compile s (first call) | full incl_h2d median / min / p90 | full excl_h2d | backbone incl / excl | power W (excl) | peak mem MiB | top-mask IoU288 / PCC masks |
|---|---:|---:|---:|---:|---:|---:|---:|
| trunk only, `mode="default"`, `fullgraph=True` (upstream's wrapping) | 17.0 | 53.703 / 53.479 / 54.292 | 50.617 / 50.393 / 51.095 | 33.954 / 31.915 | 435.3 | 4361 | 0.99986 / 0.9861 |
| trunk + fusion encoder + decoder + pixel decoder, all `fullgraph=True` (= upstream `compile=True` set) | failed | — | — | — | — | — | `Unsupported: Attempted to call function marked as skipped` — dynamo cannot trace `torch.tensor(...)` in `TransformerEncoderFusion._prepare_multilevel_features` (encoder.py:352) under `fullgraph=True` |
| trunk only, `mode="reduce-overhead"` (CUDA graphs) | 11.8 | 55.525 / 53.949 / 59.436 | 51.909 / 50.980 / 54.190 | 33.910 / 31.391 | 437.6 | 4360 | 0.99986 / 0.9861 |
| **trunk (`fullgraph=True`) + encoder + decoder + pixel decoder (`fullgraph=False`, graph breaks allowed), `mode="default"`** | 17.4 | **47.966** / 47.676 / 48.708 | **44.737** / 44.450 / 45.717 | 34.273 / 32.629 | 480.5 | 4215 | 0.99986 / 0.9852 |

Compiling the trunk alone buys ~2 ms (the 32-block ViT at 1008 is matmul/SDPA-bound already); the win comes from compiling the
small, launch-bound decoder / encoder / pixel-decoder stages (−7 ms), giving 48.0 ms incl. transfers — 1.15x the eager bf16 forward.

Other facts: model build on the CPU incl. the 3.45 GB checkpoint load 6.75 s (10.1 s in the retry run), weights -> CUDA 0.34 s,
3409 MiB allocated after load; first fp32 call 320 ms (CUDA init) then 147 ms; peak allocated 4339 MiB over all runs; idle GPU
30.5 W. In strict fp32 and in every backbone loop the GPU runs at or near its 600 W limit.

Served-like loop (the host work of `server/app.py::predict` + the GPU forward incl. readback, 5 warm + 50 iterations, medians ms; p90 of total in brackets):

| GPU precision | decode (base64 + 3.6 MB PNG decode, PIL) | preprocess (`_to_pixel_values`) | forward (full incl_h2d, text cached) | postprocess (select + 3 mask upsamples to 1920x1080 + PNG + base64) | **total** | power W | response |
|---|---:|---:|---:|---:|---:|---:|---|
| fp32 strict | 26.39 | 7.77 | 147.48 | 20.92 | **202.87** (p90 206.3) | 454.4 | 3 instances, top 0.9716 box [544.29, 447.18, 1159.29, 854.99] area 175169; 0.8199; 0.8191 |
| tf32 | 27.56 | 7.58 | 106.21 | 20.92 | **162.61** (165.0) | 360.8 | 3, top 0.9716 [544.28, 447.18, 1159.30, 854.99] area 175166 |
| bf16 autocast | 26.27 | 7.62 | 56.26 | 20.90 | **111.51** (115.0) | 281.1 | 3, top 0.9712 [544.81, 446.99, 1159.81, 854.75] area 175169 |
| fp16 autocast | 26.67 | 7.81 | 56.39 | 21.18 | **112.68** (124.7) | 282.1 | 3, top 0.9716 [544.28, 447.19, 1159.27, 855.02] area 175167 |
| p150a (PUBLISH_SUMMARY:19; DEVICE_VALIDATION served rows) | 541.5 − 486.8 ≈ 54.7 total host decode + preprocess + postprocess | | 486.8 | | **541.5** | not measured | 3 instances, top 0.9729, area 175170 (legacy graph: 0.9727, area 175127, box [545.83, 447.88, 1159.12, 854.37]) |

The host pre/post work is the same code on both sides and costs the same on this host (≈55 ms: 26 ms PNG decode of the 3.6 MB
demo image, 8 ms resize/normalise, 21 ms for three 1920x1080 mask upsamples + PNG encodes); it was measured here in this venv
(pillow 12.3) rather than in the served container. Only the `forward` term is a device comparison. The GPU response is the CPU
reference's response (top 0.9716, area 175169 vs the p150a's 0.9729 / 175170): the ranked instances, boxes (within 0.6 px) and
mask areas agree across all four precisions.

## Comparison with the p150a (matching definitions)

Ratio = p150a ms / GPU ms (> 1 means the GPU is faster). p150a precision: bf16 activations, bf8/bf16 weights on the chip
(ViT backbone: fused residual matmuls, `minimal_matmul` qkv/fc1, on-device patch embed, encoder trace), fp32 CPU neck / text
encoder / decoder / segmentation head. GPU precision per row as stated.

| row | p150a (definition) | GPU precision | GPU ms | ratio p150a/GPU |
|---|---:|---|---:|---:|
| `timing_ms.forward` = `pipeline.forward_image` (upload + device backbone & encoder + bf16 hand-off readback + CPU neck / decoder / seg head, text cached) vs GPU full forward incl_h2d (text cached) | 486.8 (PUBLISH_SUMMARY:19, Hub tt serve; 471.6-489.8 DEVICE_VALIDATION served rows) | fp32 strict | 148.753 | **3.27** |
| | | tf32 | 110.832 | **4.39** |
| | | bf16 autocast (PCC gate not met, task gates met) | 55.311 | **8.80** |
| | | fp16 autocast (PCC gate not met, task gates met) | 55.721 | **8.74** |
| | | bf16 autocast + compile (trunk fullgraph + enc/dec/pixel-dec with graph breaks) | 47.966 | **10.15** |
| GPU full forward only (excl_h2d) vs the same 486.8 | 486.8 | fp32 strict / tf32 / bf16 / fp16 | 143.118 / 105.518 / 52.487 / 52.195 | 3.40 / 4.61 / 9.27 / 9.33 |
| | | bf16 + compile (best) | 44.737 | 10.88 |
| **device backbone**: p150a backbone bench (host fp32 in, bf16 features out, fused trace) vs GPU ViT trunk incl_h2d (fp32 in, 21 MB fp32 features out) | 89.5 (DEVICE_VALIDATION EF row; README 89; island probe 90.0) | fp32 strict | 110.595 | **0.81** (p150a faster) |
| | | tf32 | 75.667 | **1.18** |
| | | bf16 autocast | 35.681 | **2.51** |
| | | fp16 autocast | 35.101 | **2.55** |
| | | bf16 autocast + compiled trunk | 33.954 | 2.64 |
| device backbone, GPU excl_h2d (no transfers) vs the same 89.5 (which includes transfers) | 89.5 | fp32 strict / tf32 / bf16 / fp16 | 111.988 / 73.335 / 33.833 / 33.665 | 0.80 / 1.22 / 2.65 / 2.66 |
| **device fusion encoder**: p150a encoder island (host in / out, trace) vs GPU encoder incl_h2d | 16.8 (DEVICE_VALIDATION EF row; README) | fp32 strict / tf32 / bf16 / fp16 | 12.452 / 11.053 / 4.047 / 4.304 | 1.35 / 1.52 / **4.15** / 3.90 |
| device fusion encoder, GPU excl_h2d | 16.8 | fp32 strict / tf32 / bf16 / fp16 | 10.888 / 9.566 / 2.641 / 2.892 | 1.54 / 1.76 / 6.36 / 5.81 |
| both device islands: p150a backbone + encoder vs GPU backbone incl + encoder incl | 106.3 | fp32 strict / tf32 / bf16 / fp16 | 123.0 / 86.7 / 39.7 / 39.4 | 0.86 / 1.23 / **2.68** / 2.70 |
| served e2e: p150a `timing_ms.total` vs GPU served-like total (same host decode / preprocess / postprocess) | 541.5 (PUBLISH_SUMMARY:19) | fp32 strict | 202.87 | **2.67** |
| | | tf32 | 162.61 | **3.33** |
| | | bf16 autocast | 111.51 | **4.86** |
| | | fp16 autocast | 112.68 | **4.81** |

Reading:

- On the part of the network the p150a runs on the chip, the eager RTX 5090 in the p150a's own precision class (bf16/fp16
  autocast) is 2.5x faster on the ViT-H backbone (35.7 vs 89.5 ms, both incl. host transfers) and ~4x faster on the fusion
  encoder (4.0 vs 16.8 ms); 2.7x on the two islands together. In strict fp32 the GPU backbone is 1.2x SLOWER than the p150a's bf16
  trace (110.6 vs 89.5 ms; the GPU sits at its 600 W limit), in tf32 1.2x faster.
- The published p150a request is dominated by host CPU work (~380 of 486.8 ms: neck, decoder, segmentation head in fp32 on the
  Ryzen). The GPU runs those stages in ~16-20 ms, so the like-for-like `timing_ms.forward` comparison is 3.3x (fp32 strict) to 8.8x
  (bf16/fp16 eager) and 10.2x (bf16 + inductor) in the GPU's favour; end-to-end with the identical host decode / mask-encode work the
  gap is 2.7-4.9x.
- Accuracy class: the GPU fp32/tf32 forwards reproduce the CPU reference (PCC ≥ 0.9997, IoU 1.0). The GPU autocast modes do not
  meet the brief's PCC > 0.999 gate on the raw 200-query logits (bf16 0.945, fp16 0.998) but produce the same served response and
  a top-mask IoU288 of 0.9997-1.0 — above the p150a's own 0.9989 / mask PCC 0.984 (its published gate). The p150a's fused default
  returns a top score of 0.9729 vs the reference's 0.9716; every GPU precision returns 0.9712-0.9717.

Not measured / not claimed: p150a power (not measured in any pass -> no power or efficiency comparison; the GPU drew 470-600 W
mean during the dense device loops, 280-455 W during the served-like loops, 30.5 W idle). p150a numbers were not re-measured;
its session-to-session served drift is ~30 ms (DEVICE_VALIDATION), the GPU's p90-median spread is < 8 ms. The GPU numbers exclude
HTTP/JSON framing, as do the p150a `timing_ms` keys. Only the demo image / prompt was run on the GPU (the port's 9-pair robustness
set was not), and only one prompt per request (the p150a's `forward_per_prompt` for 4 prompts, 482.8 ms, has no GPU counterpart
here). Pinned host memory was not used (pageable tensors, as the server produces).

## Reproduce

```bash
cd /home/deepgadget/experiments/tt-models/logs/gpu-vs-p150/sam3
HF_HUB_OFFLINE=1 /home/deepgadget/experiments/tt-models/.venv-gpu/main/bin/python bench_sam3_gpu.py --iters 50 --warmup 10 --served-iters 50 > full_run.log 2>&1
HF_HUB_OFFLINE=1 /home/deepgadget/experiments/tt-models/.venv-gpu/main/bin/python bench_sam3_gpu.py --iters 50 --warmup 10 --only-compile > compile_retry.log 2>&1
# outputs: /home/deepgadget/experiments/tt-models/reports/gpu-vs-p150/sam3.json, ./result.json, ./cpu_fp32_reference.pt
```

GPU released after the runs: `nvidia-smi --query-compute-apps=pid --format=csv,noheader` -> empty, 29 W / 668 MiB / 0 % util
(checked 2026-09-14 20:41 after the compile retry).

## Update 2026-10-03: optimized build and RTX 5090

This section compares the 2026-10-03 optimized build with the RTX 5090 rows above. The GPU measurements are unchanged. They are from 2026-09-14. The Blackhole numbers below come from an independent verification run of commit `2b0dd60` (see `VERIFICATION_2026-10-03.md`).

The measurement hardware is a Blackhole chip with a 12×10 compute grid and dispatch on ETH cores. The tt-metal tree is `8b98410e730` with `patches/tt-metal-eth-dispatch.patch`. All runs are warm, batch 1, with the demo image `media/source_1.png` and the prompt `"car"`.

The optimized build runs the whole image graph on the device as two metal traces:

- an image trace: ViT-H/14 + FPN neck + geometry encoder;
- a prompt trace: fusion encoder + decoder + scoring + seg head.

The text encoder runs on the host CPU and its output is cached per prompt. Thus, the "~380 ms of host CPU work" in the sections above does not apply to this build.

### Verified Blackhole numbers (commit `2b0dd60`)

| measurement | value |
|---|---:|
| served `/predict` `timing_ms.total`, 1 prompt (2 runs of 20 requests, median per run) | 96.8 / 96.7 ms (min 95.0 / 95.8 ms) |
| served `/predict` `timing_ms.total`, 3 prompts | 136.6 / 138.1 ms (min 136.0 / 137.2 ms) |
| served `timing_ms.forward`, 1 prompt | 82.1 / 82.4 ms |
| image trace replay | 56.36 / 56.43 ms |
| prompt trace replay | 18.27 / 18.27 ms |
| post-trace select (host selection + selected-row mask trace + readback) | 1.22 / 1.54 ms |
| `forward_image`, all 200 masks read back (n = 20) | 85.1 ms median, 84.0 ms min |
| previous release `timing_ms.forward` / `total` (rows above) | 486.8 / 541.5 ms |

In this build, `timing_ms.forward` includes the image preprocess. The fused banded input path does the resize, normalize and im2col inside the forward. Thus, `timing_ms.preprocess` is about 0 ms.

### Recomputed ratios against the existing GPU rows

Ratio = GPU ms / Blackhole ms. A value above 1 means that Blackhole is faster.

| row | Blackhole (definition) | GPU precision | GPU ms | ratio GPU / Blackhole |
|---|---:|---|---:|---:|
| served e2e: `timing_ms.total` vs GPU served-like total | 96.8 | fp32 strict | 202.87 | **2.10** |
| | | tf32 | 162.61 | **1.68** |
| | | bf16 autocast | 111.51 | **1.15** |
| | | fp16 autocast | 112.68 | **1.16** |
| served forward: `timing_ms.forward` (incl. preprocess) vs GPU served-like preprocess + forward | 82.3 (mean of 82.1 and 82.4) | fp32 strict | 155.25 | **1.89** |
| | | tf32 | 113.79 | **1.38** |
| | | bf16 autocast | 63.88 | **0.78** (GPU 1.29× faster) |
| | | fp16 autocast | 64.20 | **0.78** (GPU 1.28× faster) |
| full forward, all masks read back: `forward_image` vs GPU full incl_h2d (text cached) | 85.1 | fp32 strict | 148.753 | **1.75** |
| | | tf32 | 110.832 | **1.30** |
| | | bf16 autocast | 55.311 | **0.65** (GPU 1.54× faster) |
| | | fp16 autocast | 55.721 | **0.65** (GPU 1.53× faster) |
| | | bf16 autocast + compile (best) | 47.966 | **0.56** (GPU 1.77× faster) |

### Reading

- End to end, the optimized build is faster than the RTX 5090 in all four eager precisions. Against bf16 autocast, the lead is 1.15×.
- Most of this lead comes from the host path. This build spends about 14.5 ms on decode and postprocess (96.8 − 82.3 ms). The GPU served-like loop spends 47.2 ms on the same steps with the original PIL code (26.27 + 20.90 ms, bf16 row).
- On the model forward alone, the GPU in bf16 or fp16 autocast is about 1.3× faster. In fp32 strict and tf32, Blackhole is faster.
- The GPU host steps were not measured again with the new host code. A GPU server with the same host code would also become faster.
- The bf16 and fp16 GPU rows do not meet the PCC > 0.999 gate (see above). They meet the task gates. The Blackhole build meets its own gates: top-mask IoU288 0.99858, 3/3 instances, all-200 mask PCC 0.98064 (gate 0.98).
- Blackhole power was not measured. Thus, this section makes no efficiency comparison.