# sam3-p150: independent verification, 2026-10-03

## Verdict

**PASS.** An independent verifier measured commit `2b0dd60` again. The speed claims match within 10 %. The accuracy numbers are equal to the claimed numbers. The verifier found no cheating in the diff `20bdb6f..2b0dd60`.

## Setup

| item | value |
|---|---|
| Hardware | a Blackhole chip with a 12×10 compute grid and dispatch on ETH cores (x8 PCIe link) |
| tt-metal | `8b98410e730` with `patches/tt-metal-eth-dispatch.patch` |
| Verified commit | `2b0dd60` (the `code/` in this repo) |
| Baseline commit | `ebb55cb`. This is the earliest commit with runnable tooling. Commit `4de7fcf` is the bare import of the previous release. |
| Workload | demo image `media/source_1.png` (1920×1080 PNG), prompt `"car"`, squashed to 1008×1008, batch 1, warm runs |
| Accuracy reference | torch CPU fp32 model. The verifier generated the reference tensors again. The verifier did not use the developer's caches. |
| Host | shared host, load average 11-17 during the runs |
| Session | one session for all runs |

The verifier used these scripts: `perf/baseline_probe.py --grid 12x10 --dispatch eth --no-islands`, `perf/accuracy_suite.py --modes full`, `perf/dev/sel_phase_bench.py`, `perf/server_smoke.py` (20 requests per run). These scripts are in the source repo. They are not in `code/`.

## Measured baseline vs final

| measurement | baseline `ebb55cb` | final `2b0dd60` | change |
|---|---:|---:|---:|
| `forward_image` (all 200 masks read back), median / min | 443.4 / 441.5 ms (n = 10) | **85.1 / 84.0 ms** (n = 20) | 5.2× faster |

Round-6 A/B on the same `2b0dd60` tree. "Off" disables the 4 round-6 knobs (the round-5 code paths). "On" is the default. The runs alternate. Each cell shows two runs.

| measurement | round-6 knobs off | round-6 knobs on (default) |
|---|---:|---:|
| image trace replay | 56.74 / 56.78 ms | **56.36 / 56.43 ms** |
| prompt trace replay | 19.37 / 19.40 ms | **18.27 / 18.27 ms** |
| post-trace select | 1.37 / 1.60 ms | **1.22 / 1.54 ms** |
| served forward (`sel_phase_bench`) | 83.8 / 83.2 ms | **81.7 / 81.7 ms** |
| `server_smoke` 1 prompt `total`, median | 97.0 / 98.3 ms | **96.8 / 96.7 ms** |
| `server_smoke` 1 prompt `total`, min | 94.0 / 96.4 ms | **95.0 / 95.8 ms** |
| `server_smoke` 1 prompt `forward`, median | 83.2 / 83.8 ms | **82.1 / 82.4 ms** |
| `server_smoke` 3 prompts `total`, median | 139.9 / 140.8 ms | **136.6 / 138.1 ms** |
| `server_smoke` 3 prompts `total`, min | 136.6 / 138.0 ms | **136.0 / 137.2 ms** |

Claims on the same chip: `forward_image` 84.8 / 83.7 ms. `server_smoke` 1 prompt 98.2 -> 96.3 ms. 3 prompts 140.6 -> 138.4 ms. The verified numbers are within 10 % of each claim.

## Accuracy gate results

Demo image, `"car"`, `baseline_probe`:

| metric | gate | `2b0dd60` | baseline `ebb55cb` |
|---|---:|---:|---:|
| top-mask IoU288 | ≥ 0.997 | **0.998576** | 0.99886 |
| instances (reference 3) | same set | **3/3**, same top index | 3/3 |
| top score (reference 0.97158) | within about 0.002 | **0.97306** | 0.97286 |
| all-200 mask PCC | ≥ 0.98 | **0.980642** (pass) | 0.98443 |
| logits PCC | - | 0.96971 | 0.9584 |
| boxes PCC | - | 0.98659 | 0.9850 |

- These numbers are exactly equal to the claimed numbers.
- `accuracy_suite --modes full` on 6 images: every image has the same instance set as the reference. Top IoU is 0.9964-0.9987.
- The suite JSON "full" block is exactly equal to the round-4 result (`opt_logs/r4/suite_final_chip30.json`). This confirms that the device output bits did not change since round 4.
- `server_smoke` gives the same results with the round-6 knobs off and on: car 3 instances, person 1, tree 5. The scores and mask areas are identical.
- Host tests: 44/44 pass.

## Disclosed numerics changes

- Round 6 does not change numerics. Every round-6 change keeps the device output bits (6-image dumps, selected == full).
- The earlier rounds changed numerics. The all-200 demo mask PCC went from 0.98443 (baseline) to 0.98064. The cause is from earlier rounds: the device neck, decoder and seg head, the custom GELU kernel, the GroupNorm rewrite and the SDPA chunk sizes. [`OPT_REPORT.md`](OPT_REPORT.md) discloses each change.
- The result still passes the 0.98 gate, but the margin is very small.
- One numerics experiment was rejected in round 6. A bf16 image upload dropped the demo all-200 mask PCC to 0.97802 (below the gate). It also changed the "person" instance set on one suite image. The code was reverted.

## Review findings

- The demo all-200 mask PCC is 0.98064 against a 0.98 gate. The baseline had 0.98443. The margin is very small. This regression is from earlier rounds, not from round 6, and it is disclosed.
- The 1-prompt served `total` gain of round 6 is small: about 97.6 -> 96.7 ms median over 2 pairs. This is close to the host noise. The gains in the forward (-1.2 ms) and in the prompt trace (-1.1 ms) are clear.
- `perf/dev/sel_phase_bench.py` sets `sys.path` to the main repo's `code/` and not to its own tree. This had no effect here, because the main tree was clean at `2b0dd60`.
- No cheating in the diff `20bdb6f..2b0dd60`. Every change is an op or configuration swap with unchanged output bits. The mask rows are still computed on the device. The kept rows go through a one-hot matmul. The full readback still computes all 224 rows. The timing includes all of this work.
- During round 6, one device hung in an isolated micro-benchmark (`perf/dev/rpb_mask_l1_test.py`). The model code does not contain that benchmark's op sequence. On a second chip, all 6 image dumps were bit-identical to the first chip. [`OPT_REPORT.md`](OPT_REPORT.md) ("Known hangs (round 6)") gives the details.

## Update 2026-10-04

Two later commits change `code/`. Commit `6080ef8` adds the image cache and `POST /predict_raw`. Commit `54d1f2a` adds the Python API (`tt_sam3.Sam3`). Independent verifiers measured both commits again. The device programs and the output bits do not change. The device numbers and the accuracy numbers above stay the same.

### Image cache and `/predict_raw` (commit `6080ef8`, chip 7, verified against base `2b0dd60`)

| measurement | base `2b0dd60` | `6080ef8`, cache off | `6080ef8`, cache on |
|---|---:|---:|---:|
| `forward_image` median / min | 99.4 / 95.7 ms | 97.3 / 95.0 ms | – |
| server_smoke `timing_ms.total`, 1 prompt, same image | 97.8 ms | 97.0 ms | **22.2 ms** |
| server_smoke `timing_ms.total`, 3 prompts, same image | 138.1 ms | 137.9 ms | **67.5 ms** |
| HTTP client wall, JSON `/predict`, same request repeated | 114.6 ms | 111.7–114.3 ms | **34.4 ms** |
| HTTP client wall, `/predict_raw` PNG body | – | 102.2 ms | 26.3 ms (repeated) |
| HTTP client wall, `/predict_raw` raw RGB24 body | – | 92.8 ms | 28.9 ms (repeated) |

- A request with a new image costs the same time with the cache on or off.
- DUMP_ALL dumps of all 200 masks, logits, boxes and presence on the 6 suite images are bit-identical between `2b0dd60` and `6080ef8`.
- baseline_probe gives IoU288 0.998576, mask PCC 0.980642 (gate 0.98) and top score 0.973065 for both commits.
- A fixed sequence of 30 HTTP requests gives the same results sha256 for the base, cache off and cache on. The sequence includes cache hits, misses and invalidation.
- `/predict_raw` gives the same results as `/predict`. The `/predict` error codes are the same as in the base.
- Host tests: 56/56 pass.

### Python API (commit `54d1f2a`, chip 14, ETH dispatch, 12×10 grid, fresh venv)

`SAM3_NUM_THREADS=8`, 3 interleaved rounds of 10 calls each, warm, demo image:

| call | median | min |
|---|---:|---:|
| `model(rgb, "car", return_masks=False)` | 82.75 ms | 80.38 ms |
| `pipeline.forward_rgb_selected_multi` called directly | 82.59 ms | 80.10 ms |
| Published model-call number | 82.32 ms | 80.82 ms |
| `model(rgb, "car")` with masks | 84.63 ms | 81.85 ms |
| `model(path)` (includes the PNG decode) | 96.97 ms | 95.90 ms |
| Repeated image (image cache) | 20.78 ms | 20.65 ms |
| 3 prompts on a new image | 124.61 ms | 124.08 ms |

- The API median is 0.5 % above the published median. The API min is lower than the published min.
- With the default thread count, the call gives 81.01 ms median and 80.48 ms min over 30 calls.
- `test_api_device.py`: 2 passed. The API output is bit-identical to the server handler and to `forward_rgb_selected_multi`. The demo results are equal to the published results (car 0.9731 / 0.8171 / 0.7969, person 0.5936, tree 5 instances). All input types and the image cache give the same bits.
- A real uvicorn server and a separate Python process give the same scores, boxes, presence, mask PNGs and mask areas for `car`, `person` and `tree` on the demo image.
- HTTP smoke test: the first warm-up takes 112.9 s and the second takes 0.09 s. One `/predict` with 3 prompts takes 284 ms client wall (first request). The server stops cleanly on SIGINT.
- `test_api_host.py`: 18 passed.
- The diff `6080ef8..54d1f2a` has no device or numerics change. The only change under `tt/` is in `mm_fused.py`, which now finds the kernel directory from `TT_METAL_HOME` or the `ttnn` tree.