# sam3-p150: optimization baseline on a Galaxy Blackhole chip (chip 8) Date: 2026-10-02. Hardware: UMD chip 8 of the 32-chip BH Galaxy (BDF 0000:41:00.0, /dev/tenstorrent/20). tt-metal: shared tree at `8b98410e730` with the ETH-dispatch patch. Model code: the baseline commit `4de7fcf`, unchanged. These are baseline measurements only. No model code was changed; the probe and profiler scripts are new files in `perf/`. ## 0. Setup (reproducible) ```bash ROOT=/home/ttuser/experiments/tt-models; M=$ROOT/models/sam3-p150 $ROOT/tools/mkvenv.sh $M # .venv layered on the shared python_env $M/.venv/bin/python -m pip install -c $M/.venv/my-constraints.txt \ 'sam3==0.1.4' iopath psutil 'triton==3.6.0' einops pycocotools ftfy regex tqdm # my-constraints.txt pins torch==2.11.0+cpu, numpy==1.26.4, torchvision==0.26.0+cpu. The generated # shared-constraints.txt came out empty because pip freeze does not work in the uv env. # weights (gated): facebook/sam3 sam3.pt + config.json @ 3c879f39826c281e95690f02c7821c4de09afae7 -> ~/.cache/huggingface ``` Every run uses: `source $ROOT/tools/chipenv.sh 8 $M/.venv; export HF_HUB_OFFLINE=1 SAM3_TORCH_COMPILE=0 SAM3_NUM_THREADS=17`. chipenv.sh now always exports `TT_GRID=12x10`, which sets `TT_METAL_CORE_GRID_OVERRIDE_TODEPRECATE=11,9`. That is the policy: the target is the p150 12x10 grid, and 13x10 is forbidden. The 11x10 runs below (p150 with Tensix dispatch, which is what the authors measured) override the variable to `10,9` inside the process. They are measurements only. | what | command (cwd = `$M/code` unless noted, `PYTHONPATH=$M/code:$PYTHONPATH`) | |---|---| | backbone PCC + speed (the author's harness) | `TT_WEIGHTS_REVISION=3c879f39826c281e95690f02c7821c4de09afae7 python -m tt_sam3.benchmark --mode backbone --pretrained --iters 20 --warmup 3` | | e2e harness (random input, prompt "object") | same with `--mode e2e --iters 10 --warmup 2` | | **demo accuracy (README row) + stage split + island split** | cwd `$M`: `python perf/baseline_probe.py --grid 12x10 --dispatch worker --iters 10 --ref-cache /ref_car.pt --json baseline_logs/probe_12x10_worker.json` (use `--dispatch eth` to open with ETH dispatch through tools/open_device_12x10.py, and `--grid 11x10` for the p150-equivalent grid) | | device profiler (per-op device time) | cwd `$M`: `TT_METAL_DEVICE_PROFILER=1 TT_METAL_PROFILER_CPP_POST_PROCESS=1 TT_METAL_PROFILER_DISABLE_PUSH_TO_TRACY=1 TT_METAL_PROFILER_NO_CACHE_OP_INFO=1 TTNN_OP_PROFILER=1 python perf/profile_islands.py`, then `python perf/analyze_devprof.py $TT_METAL_PROFILER_DIR/.logs/cpp_device_perf_report.csv`. Do not use `python -m tracy`: its post-processing got killed after parsing an 850 MB host CSV, and the tracy capture ports are shared with other agents. | | host-only unit tests | `python -m pytest -q tt_sam3/tests/test_fused_host.py` | ## 1. Accuracy (reproduced) Demo image `media/source_1.png`, prompt "car", fused default path (TT_FUSED=1), compared with the torch CPU fp32 reference computed in the same process: | metric | README (p150a) | chip 8, 11x10 | chip 8, 12x10 Tensix dispatch | chip 8, 12x10 ETH dispatch | |---|---|---|---|---| | top-mask IoU288 vs CPU fp32 | 0.999 (0.9989) | 0.99886 | 0.99886 | 0.99886 | | top score TT / ref | 0.973 / 0.972 | 0.97286 / 0.97158 | same | same | | instances > 0.5 TT / ref | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | | `pred_masks` PCC (200x288x288) | 0.984 | 0.98444 | 0.98444 | 0.98444 | | `pred_logits` / `pred_boxes` PCC | - | 0.9584 / 0.9850 | same | same | | ViT backbone PCC (benchmark, random input, pretrained) | 0.993 / 0.9915 | 0.99180 | 0.99180 | - | | e2e harness `mask_pcc` / `score_pcc` (random noise, "object") | - | 0.9039 / 0.6717 | 0.9039 / 0.6717 | - | Gates for later steps, as the authors used them: top-mask IoU288 >= 0.997, the same 3 instances, top score within about 0.002 of the reference, mask PCC >= 0.98 on the demo, and backbone PCC >= 0.99. Whole-output PCC on random noise is not a meaningful gate; DEVICE_VALIDATION notes that the legacy path itself scores 0.93-0.97 there. Tests: `test_fused_host.py` 17/17 passed. `test_ttnn_sam3_vit_backbone.py` passed. `test_ttnn_sam3_transformer.py` errors because the `sam3_transformer` fixture is missing. In `test_ttnn_sam3_e2e.py`, 1 test fails because it calls the removed method `pipeline.run_vit_backbone`. Both failures were already in the baseline test suite; they are not device problems. ## 2. Performance baseline All numbers are warm medians. The host has 64 shared cores, about 11 agents and a load average around 20, so host-stage numbers are noisy: forward min/max ranged 346-551 ms. | measurement | README p150a | chip 8 11x10 (Tensix dispatch) | chip 8 12x10 (Tensix dispatch) | chip 8 12x10 (ETH dispatch) | |---|---|---|---|---| | `forward_image` demo, served path (incl. host stages) | 472-490 ms (~480) | 464.6 / 490.9 ms (2 runs) | 434.9 ms | 456.6 ms | | e2e harness `avg_ms` | - | 453.5 ms | 433.0 ms | - | | backbone harness (host in -> host out) | 89.5 ms | 98.8 ms | 97.6 ms | - | | backbone device graph, trace replay (device only) | (90.0 island) | 84.39 ms | 84.27 ms | 83.88 ms | | backbone device graph, eager with sync | - | 84.54 ms | 84.06 ms | 83.85 ms | | fusion encoder trace replay (device only) | 16.8 (island, incl. I/O) | 13.76 ms | 13.62 ms | 13.65 ms | | device-profiler kernel sum per forward | - | 95.8 ms (backbone 82.2 + encoder 13.6) | 94.7 ms (81.2 + 13.5) | - | Takeaways: - Device work is about 98 ms out of a 435-490 ms forward, roughly 20 %. **About 340-390 ms per forward is host CPU torch.** - **Going from 11x10 to 12x10 does not change backbone or encoder device time** (84.4 vs 84.3 ms). The matmul and SDPA configs were tuned for 11x10 and do not use the extra column (see section 5). Elementwise ops (gelu, RoPE) do scale: about -8 %. - ETH and Tensix dispatch give the same device time on this Galaxy, because both have 12x10 after the cap. ETH dispatch matters for a p150, where it frees the 12th column. - The backbone trace replay equals the eager run: with ops averaging 0.2 ms, host dispatch is hidden. Tracing the backbone alone saves almost nothing today, but it becomes necessary once ops get small. - The backbone harness reads 98 ms here against 89.5 ms on the p150 because of the x1 PCIe link (H2D 3.9 ms + D2H 8.5-9.8 ms, against about 1-2 ms on x16). See section 3. ## 3. Host vs device breakdown (demo forward, 12x10 Tensix dispatch; 11x10 run in parentheses) | stage | where | ms (median) | notes | |---|---|---|---| | ViT-H trunk (`tt_vit_backbone_fused`) | device + I/O | 96.8 (98.5) | = im2col 0.1 + from_torch 0.5-0.7 + **H2D 3.9** + device 84 + **D2H 8.5-9.8** | | FPN neck `convs[0]` (2x deconv 2x2, gelu, 1x1, 3x3 @288²) | host bf16 | 79-84 | the largest single host op | | FPN neck `convs[1]` (deconv, 1x1, 3x3 @144²) | host bf16 | 24-28 | | | FPN neck `convs[2]` (1x1, 3x3 @72²) | host bf16 | 5 | | | FPN neck `convs[3]` (maxpool, 1x1, 3x3 @36²) | host bf16 | 2 | **computed, then thrown away** (`scalp=1` drops the lowest level) | | `backbone.forward_image` total | | 205-218 | trunk + neck + pos-enc (cached) | | text encoder | host fp32 | 0 (cached) | **~93 ms on every new prompt** (CLIP-style, 354 M params). The cache is keyed by prompt. | | geometry encoder (CLS token cross-attends 5184 image tokens) | host | 12-14 | depends on the image, so it cannot be cached across images | | fusion encoder `_run_encoder` | host prep + device | 19.8-20.7 | `TtFusedEncoder.__call__` 19.1-19.6 = from_torch 2.4-3.0 + H2D 0.84 + trace 13.6 + D2H+float 2.2-2.7. Host prep (pool text, flatten) is about 1 ms | | transformer decoder (6 layers, 201 queries, box-RPB) | host bf16 autocast | 86-98 | | | segmentation head | host bf16 | 109-134 | pixel decoder (2x conv3x3 + GN + upsample to 288²) **74-91**; mask predictor and the other heads are the rest | | misc (scoring, box head, ...) | host | ~5 | | | **forward total** | | **435-491** | device about 98 ms; PCIe transfers about 17 ms; the rest is host torch | ### Every host<->device transfer per forward (steady state, same prompt) | # | dir | tensor | size | time on chip 8 (x1 Gen5) | |---|---|---|---|---| | 1 | H2D | backbone im2col input, fp32 ROW_MAJOR `[9,576,608]` | 12.6 MB | from_torch 0.5 ms + **3.9 ms** | | 2 | D2H | backbone features, bf16 `[1,72,72,1024]` (passed to the CPU neck as a channels_last view) | 10.6 MB | **8.5-9.8 ms** (~1.2 GB/s D2H) | | 3 | H2D | encoder input tokens, bf16 ROW_MAJOR `[1,5184,256]` (persistent trace input) | 2.65 MB | from_torch 2.4-3.0 ms + 0.84 ms | | 4 | D2H | encoder memory, bf16 `[1,5184,256]` -> fp32 | 2.65 MB | 2.2-2.7 ms | | (5) | H2D | per-prompt text K^T/V (6 layers) + key-padding bias | ~0.2 MB | only when the prompt changes (cached on device) | That is about 17-20 ms of transfer and conversion per forward on Galaxy, against roughly 3-5 ms on a p150's x16 link. All of it disappears if the neck, decoder and segmentation head move to the device. Only the input upload and the final outputs would remain. ## 4. What is fused / traced already, op count The TT_FUSED=1 default is the authors' 2026-09-13 "megakernel pass". It is a partial fusion, not a megakernel: - **Backbone, eager, not traced: 391 device ops per forward.** Ingest is tilize + patch-embed matmul with pos fused (`dit_minimal_matmul_addcmul_fused`, HiFi4 fp32-acc) + ln_pre. Then 32 blocks x 12 ops each: LN, qkv `minimal_matmul`+bias, `split_query_key_value_and_split_heads`, RoPE q, RoPE k, SDPA, concat heads, proj matmul+bias+residual (fused), LN, fc1 `minimal_matmul`+bias, **separate exact gelu**, fc2 matmul+bias+residual (fused). Window-order residency is used: no per-block partition, global blocks run on a zero-copy merged view, and there is one final unpartition (3 reshape + 1 permute = 0.77 ms). - **Fusion encoder: 1 metal trace, 115 device ops** (1 tilize + 6 x 19). The pos term is folded into a per-layer constant for the fused qkv matmul. The text K/V is precomputed on the host per prompt. It uses scale_mask_softmax_in_place, matmul+residual fusion and linear+relu. - **Total: 506 device ops per forward**, plus 2 H2D and 2 D2H. - Not on device at all: the FPN neck, text encoder, geometry encoder, decoder and segmentation head. `tt/ttnn_sam3_neck.py`, `ttnn_sam3_segmentation.py` and `ttnn_sam3_text_encoder.py` are CPU-torch stubs used only by the tests. ### Device-time sinks (device profiler, 12x10, kernel durations; `baseline_logs/device_profile_12x10_worker.txt`) Backbone, 81.2 ms over 391 ops; op-to-op gaps total only 0.2 ms: | op class | n | ms | % | notes | |---|---|---|---|---| | MinimalMatmul (qkv / proj+res / fc1 / fc2+res / patch) | 129 | 32.35 | 39.8 | about 130-140 TFLOP/s on qkv/fc1; per call qkv 0.229, proj+res 0.125, fc1 0.313, fc2+res 0.341 ms; **does not scale 110->120 cores** (31.96 -> 32.35 ms) | | **Unary (exact gelu after fc1)** | 32 | **13.77** | **17.0** | 0.43 ms per call, **more than the fc1 matmul itself** (0.31 ms) | | SDPA | 32 | 12.17 | 15.0 | windowed 0.247 ms per call (9x16 heads x 576, about 49 TFLOP/s); global 1.31 ms per call (5184², about 84 TFLOP/s); q/k chunk 288 | | RotaryEmbedding (q and k separately) | 64 | 8.74 | 10.8 | 0.137 ms per call | | LayerNorm (DRAM interleaved) | 65 | 5.76 | 7.1 | 0.088 ms per call on 10.6 MB | | NlpCreateHeads | 32 | 5.49 | 6.8 | 0.172 ms per call | | NLPConcatHeads | 32 | 1.99 | 2.5 | | | final unpartition (reshape x3, permute) | 4 | 0.85 | 1.0 | | Encoder, 13.5 ms over 115 ops: **SDPA 6.72 ms (50 %**; S=5184, 8 heads, head_dim 32, q/k chunk 64, about 25 TFLOP/s), MinimalMatmul 1.62, **ReshapeView 1.55** (`reshape(ca_q,[1,S,8,32])` is a real tiled copy, 0.26 ms per layer), Matmul on 81 cores 1.36, relu+ff1 Unary 0.64, LN 0.51, Transpose 0.35. ## 5. Core-grid usage, and where grid assumptions live - **11x10 is not hard-coded anywhere.** Every program config takes `device.compute_with_storage_grid_size()`: `ttnn_sam3_vitdet.py:735-743` (MinimalMatmulConfig for MM and DIT), `:813` (backbone SDPA), legacy `:115`; `ttnn_sam3_encoder.py:154, 355-363`. - The **block sizes are constants tuned on the p150's 11x10 grid**: `fused_config.py` has `DEFAULT_MM_BLOCKS=(8,8,4,4,2)` and `DEFAULT_DIT_BLOCKS=(4,4,4,2,2)` (the second taken from rf-detr); SDPA chunks are 288/288 for the backbone and 64/64 for the encoder. With M=5184 rows (162 tiles) split over 12 columns, 13.5 tiles per core round up to 2 M-blocks of 8 = 16, which is 84 % utilisation; on 11 columns it is 14.7/16 = 92 %. N per row: qkv 96/10, fc1 128/10 and fc2 32/10 tiles against N_block 4 also lose about 20 %. This is the likely reason matmul time does not drop on 12x10. It has not been verified yet. - Ops that do not use the full grid: tilize (81 cores) and `ttnn.matmul` in encoder cross-attention (81 cores, auto config). - With default Tensix dispatch the Galaxy chip already gives 12x10. ETH dispatch gives the same 12x10 (capped) and costs about 0.9 us per traced op (about 0.1 ms for the 115-op encoder trace). The model uses 1 CQ, so ETH dispatch is compatible. `device_open_kwargs` (`fused_config.py:150`), `server/app.py:174`, `benchmark.py` and `conftest.py` all open with `ttnn.open_device(device_id, trace_region_size=90 MB)`, which is Tensix dispatch. To run on a p150 with ETH dispatch, add `dispatch_core_config=DispatchCoreConfig(ETH)` there. ## 6. Ranked optimization opportunities Estimates are per forward, demo request, 12x10, with the gain on a p150 in parentheses where it differs. Host-stage gains assume the device replacement costs what similar ops cost in section 4. 1. **Move the FPN neck to the device (convs[0..2]); stop computing convs[3], which `scalp=1` throws away.** The 2x2/s2 deconvs are pure matmuls ([N,1024]x[1024,4*512] + pixel-shuffle). The 1x1 convs are matmuls; the 3x3 convs use `ttnn.conv2d`; gelu can be fused. **Gain: -110 to -118 ms host, plus the 10.6 MB backbone D2H (-8.5 to -9.8 ms on Galaxy, about -1.5 ms on p150).** Expected device cost is about 3-6 ms; the 3x3 at 288² is 49 GFLOP. This also removes the encoder H2D (#3), because level-2 features and pos stay on the device. Risk: the neck runs in bf16 today, so accuracy should hold. 2. **Move the segmentation head to the device** (pixel decoder: 2x conv3x3 + GroupNorm + nearest/bilinear upsample; mask predictor MLP + einsum `[200,256]x[256,288²]`; cross_attend_prompt; instance and semantic heads). **Gain: -105 to -130 ms host** at a device cost of about 3-6 ms. New D2H: `pred_masks` 200x288x288 bf16 = 33 MB, about 27 ms on Galaxy x1 and about 3 ms on p150. The outputs must not be truncated: all 200 masks stay in the output contract. 3. **Move the transformer decoder to the device** (6 layers, d=256, 201 queries with the presence token; self-attention, text cross-attention, image cross-attention with the box-RPB log-delta bias recomputed per layer from the reference boxes; FFN; box refinement). **Gain: -85 to -98 ms host** at a device cost of about 5-10 ms. This is the hardest port because of the per-layer RPB mask `[8,201,5184]` and the iterative boxes; all of it can stay on device inside one trace. 4. **One trace for the whole image graph** (backbone + neck + geometry encoder + fusion encoder + decoder + segmentation head) once items 1-3 land. That means one H2D and one D2H per forward. The geometry encoder (-12 to -14 ms host) belongs here because it is 1 query over 5184 keys and nearly free on device. Tracing also removes eager dispatch for the hundreds of small decoder ops; untraced they would cost 60-190 us each under host load. 5. **Backbone: fuse or replace the separate exact gelu** (13.8 ms, 17 % of the backbone). Options: an activation fused into `minimal_matmul` (rf-detr found it slower, but this has not been measured here), `ttnn.linear(..., activation="gelu")`, or an approximate gelu gated on the IoU/PCC gates. **Gain: -8 to -13 ms device.** 6. **Backbone: retune MinimalMatmulConfig and SDPA chunks for 12x10** so the 12th column is used: M_block such that 162/12 fits (for example 7 or 14), N_block per N (96/128/32 tiles over 10 rows), and windowed SDPA q_chunk 96/192 to fix 288 work units over 120 cores. **Gain: -5 to -9 ms device** (matmul 32 ms at about 80 % tile utilisation; windowed SDPA 6.9 ms, global 5.3 ms). 7. **Backbone: RoPE + head-split fusion.** Two RotaryEmbedding calls (8.7 ms) + NlpCreateHeads (5.5 ms) + concat (2.0 ms) = 16.2 ms of data movement. Options: one RoPE call on a stacked q|k head tensor, a create-heads variant that writes the rotated layout, or sharded L1 versions. **Gain: -4 to -8 ms device.** 8. **Encoder SDPA tuning** (6.7 ms, q/k chunk 64 at head_dim 32; try 256/512) and replacing the cross-attention `reshape+permute` with a head-split op (1.55 + 0.35 ms). **Gain: -4 to -6 ms device.** 9. **Input upload:** upload uint8 or bf16 pixels (3-6 MB) and do normalize + im2col on device, instead of 12.6 MB of host-side fp32 im2col. **Gain: -2 to -3 ms on Galaxy, under 0.5 ms on p150.** The patch embed currently uses fp32 + HiFi4 for accuracy; bf16 input needs a gate check. 10. **LayerNorm sharding / fusion** (65 LNs, 5.8 ms DRAM-interleaved) and L1-sharded residual stream between ops. **Gain: -2 to -4 ms device.** 11. **Served path, not in the single-prompt benchmark:** a request with k prompts reruns the whole image graph k times (README caveat). Run backbone + neck once per image and only encoder/decoder/head per prompt, or batch the prompts. This is computation reuse within one request, not caching outputs across runs. **Gain: about -(k-1) x 200-300 ms per k-prompt request.** A cold prompt also costs about 93 ms of CPU text encoder; putting the text encoder on device would bring that to a few ms. Projected end state if items 1-4 land: about 100-130 ms device, plus about 30 ms of I/O on Galaxy (about 5 ms on p150), against about 435-490 ms today. Items 5-10 then take the device part to roughly 60-75 ms. The CPU work, about 80 % of today's forward, is the main target. Device-only optimisation (items 5-10) is bounded at about -35 ms out of about 98 ms. ## 7. Notes / blockers - None blocking. Weights download worked with the shared HF token; the chip was stable through about 10 device runs. - `python -m tracy` is unreliable on this shared host: the default ports collide between agents, and post-processing was killed. Use the device-profiler environment variables plus `perf/analyze_devprof.py` instead. - Host contention: the pipeline forces `torch.set_num_threads(17)` (SAM3_NUM_THREADS), which overrides chipenv's OMP=5. Host-stage timings swing ±15 %. - Files: `perf/baseline_probe.py`, `perf/profile_islands.py`, `perf/analyze_devprof.py`, and `baseline_logs/` (probe JSON/TXT, harness logs, device-profiler CSVs and summaries for 11x10 and 12x10). - Post-crash check (2026-10-02 11:17 UTC): after the unexpected host shutdown, chip 8 opened cleanly with no reset needed. The `open_device_12x10.py 12x10` self-test gave trace PCC 0.99967 at 0.384 ms/iter. The baseline numbers above come from commit `ebb55cb` and were measured before the crash. Commit `37a4a8c` is an UNVERIFIED snapshot of later optimisation work: `ttnn_sam3_device_graph.py` and `perf/dev/*`. It is not part of this baseline, and none of the numbers above depend on it. - Post-reboot re-check #2 (2026-10-02 16:22-16:40 UTC, after the second host crash and reboot). Chip 8 opened cleanly with no reset; the `open_device_12x10.py 12x10` self-test gave trace PCC 0.99967 at 0.384 ms/iter. **The /dev/tenstorrent numbering changed across the reboot**: chip 8 (BDF 0000:41:00.0) is now `/dev/tenstorrent/24`, not 20. Always take `TT_CHIP_PCI_DEV` from chipenv.sh before any `tt-smi -r`; never reuse a recorded number. `perf/baseline_probe.py --grid 12x10 --dispatch worker --iters 10` was re-run on the baseline code (a worktree at `ebb55cb`, i.e. model code `4de7fcf`). Logs are in `baseline_logs/probe_12x10_worker_postreboot2.{txt,json}`. The accuracy matched to every printed digit: IoU288 0.99886, top score 0.97286 / 0.97158, 3 / 3 instances, mask PCC 0.98444, logit / box PCC 0.9584 / 0.9850. Device time also matched: backbone trace 83.42 ms (eager 83.50), encoder trace 13.68 ms, H2D 3.90 ms, backbone D2H 8.41 ms. The forward median was lower, 357.3 ms (min 331, max 414), against 435 ms before. The host was nearly idle (load average 0.5, with the other agents not yet restarted), so the host torch stages ran faster: FPN conv[0] 44 ms against 79-84, decoder 73 against 86-98, segmentation head 83 against 109-134. This shows how much the host-stage numbers depend on load. The device numbers in sections 2-4 are load-independent. - Post-reboot check #3 (2026-10-03 00:45 UTC, after host crash #3 on 2026-10-02 at 16:40). **No device work was run on chip 8 during this resume.** Crash #3 began with a device hang on chip 8. That hang came from post-baseline optimisation work, which was not part of the baseline code; see `OPT_REPORT.md`. A single retry then failed with an ARC/MMIO timeout, and the chip dropped off PCIe in an AER storm at root port 0000:40:01.1. Per the hang rule ("retry once; if still broken, stop and report"), chip 8 is not re-opened. After the reboot the link is back, at 32 GT/s, and the FAULT marker was renamed to `FAULT.2026-10-02_1637`. But re-opening a chip that took the host down is a risk to all 11 agents, so it needs an explicit go-ahead from the orchestrator or user. A different chip would also need to be assigned explicitly. Checks that did not touch the device: `git fsck --full` is clean, with one dangling commit and no corruption; `test_fused_host.py` passes 17/17 with `SAM3_FULL_DEVICE=1` and with `=0`; the `.venv` is intact. The baseline numbers in sections 1-6 still stand. They were measured twice, before crash #2 and after reboot #2, with identical accuracy and device time.