# Serving SAM3 on Blackhole with tt-model-manager

This repo is a complete **tt-model container package source**: `tt-model.yaml` (the
manifest), `code/tt_sam3` (the port, incl. the ASGI server `tt_sam3.server.app:app`),
`media/` (demo inputs) and this file. `tt-model package --container` builds the image from
it; `tt-model serve` / `tt serve` run it; `tt-model push` publishes it.

Hardware target: exactly one Blackhole **p150a** (`hardware: p150`, `mesh_device: P150`).
tt-metal tree: `/home/deepgadget/experiments/gbp-tt/tt-metal` (main `8b98410e730`,
`v0.78.0-dev20260820-25`, torch 2.11.0 / torchvision 0.26.0). Weights: `facebook/sam3`
@ `3c879f39826c281e95690f02c7821c4de09afae7` (`sam3.pt`, gated -- Meta SAM License).

## 1. Running on the HOST (validation, no Docker)

Prerequisites: the tt-metal tree above built with its `python_env` (py3.10 here; the image
uses py3.12 -- both work), the card visible at `/dev/tenstorrent/0`, `hf auth login` done
with an account that has access to `facebook/sam3`.

```bash
ROOT=/home/deepgadget/experiments/tt-models
REPO=$ROOT/models/sam3-p150
TREE=/home/deepgadget/experiments/gbp-tt/tt-metal
PY=$TREE/python_env/bin/python

# 1. serve-path packages the tree venv lacks -- install them into a THROWAWAY overlay dir,
#    never into the tree venv (torch/torchvision stay the tree's 2.11.0 / 0.26.0):
uv pip install --python $PY --target /tmp/sam3-overlay \
    fastapi uvicorn 'pydantic>=2' 'sam3==0.1.4' iopath psutil 'triton==3.6.0' einops \
    pycocotools ftfy regex tqdm typing_extensions 'timm>=1.0.17' huggingface_hub 'numpy>=1.26,<2'
# (timm/huggingface_hub/numpy are already in the tree venv; listing them is harmless.)

# 2. pre-fetch the pinned checkpoint into the HF cache (3.45 GB; once):
hf download facebook/sam3 sam3.pt config.json --revision 3c879f39826c281e95690f02c7821c4de09afae7

# 3. run uvicorn exactly the way the container does
export PYTHONPATH=$REPO/code:$TREE:$TREE/ttnn:$TREE/tools:/tmp/sam3-overlay
export TT_METAL_HOME=$TREE TT_METAL_RUNTIME_ROOT=$TREE
export HF_MODEL=facebook/sam3
export TT_WEIGHTS_REVISION=3c879f39826c281e95690f02c7821c4de09afae7
export TT_MESH_SHAPE=1x1 MESH_DEVICE=P150 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0
export SAM3_TORCH_COMPILE=0          # 1 only if a C++ compiler is present (inductor)
export SAM3_BPE_PATH=$REPO/code/tt_sam3/assets/bpe_simple_vocab_16e6.txt.gz   # optional: this is the default
cd $REPO && $PY -m uvicorn --host 0.0.0.0 --port 20000 --lifespan on tt_sam3.server.app:app
```

The log shows `Loading weights ...`, `Loading checkpoint ...`, `Opening device ...`,
`Loading pipeline ...`, `Warming up (1/2) ...`, `Warmup complete`, then uvicorn's
`Application startup complete` -- only then is the server ready. Cold boot = 3.45 GB
`torch.load` (~1 min) + weight conversion (seconds) + ttnn JIT of the kernels (minutes; cached
under `TT_METAL_CACHE`, default `~/.cache/ttnn`). Stop with Ctrl-C / SIGTERM: the lifespan
shutdown drops the device tensors and calls `ttnn.close_device`.

Offline host run: `export SAM3_WEIGHTS_DIR=/dir/containing/sam3.pt` skips the Hub entirely.

Smoke test (the hardware phase runs this unchanged):

```bash
python3 $REPO/code/tt_sam3/server/smoke_test.py --url http://127.0.0.1:20000 --out /tmp/sam3_overlay.png
# PASS prompt='car' instances=N top_score=0.97x area_frac=0.0xx inside_box=99.x% box=[...] forward_ms=... total_ms=...
```

It posts `media/source_1.png` with prompt `car` and requires: `/health` ok, >= 1
instance, top score > 0.5, top mask non-empty and covering 0.1%..60% of the image,
>= 95% of its pixels inside the returned box, box inside the image. The overlay PNG is
for eyeballing against `media/target_1_ttnn.png`.

Environment knobs the server reads (all in the lifespan, none at import): `HF_MODEL`,
`TT_WEIGHTS_REVISION`, `SAM3_WEIGHTS_DIR`, `TT_MESH_SHAPE` (`1x1` | `(1, 1)` | `1,1`;
anything else is refused), `TT_DEVICE_ID`, `SAM3_BPE_PATH`, `SAM3_TORCH_COMPILE` (default
0), `SAM3_NUM_THREADS` (default 17, capped at `os.cpu_count()`), `SAM3_WARMUP_RUNS` (2),
`SAM3_L1_SMALL_SIZE` (0 = ttnn default, the validated E2E setting; the PCC tests used 32768),
`SAM3_MAX_PROMPTS` (4), `SAM3_MAX_SIDE` (4096), `SAM3_TEXT_CACHE_MAX` (256).

`TT_FUSED` (default **on** since the device pass of 2026-09-13; `serve.env` sets `"1"`
explicitly, `TT_FUSED=0` = the legacy device graph) runs the two device islands as the fused
paths of the megakernel pass: window-order residency, `minimal_matmul` for qkv/fc1 and
`dit_minimal_matmul_addcmul_fused` for proj/fc2 in the ViT backbone, on-device patch embed;
the fusion encoder as one metal trace with fused ops and per-prompt text K/V kept on device.
The server then opens the chip with `trace_region_size=SAM3_TRACE_REGION_SIZE` (90 MB) and
refuses to report READY unless warm-up run 1 captured the encoder trace and every warm-up
forward ran through it (`GET /info` → `runtime.encoder_trace_captured`, `runtime.encoder_fallbacks`
must be 0; the log shows `Encoder trace captured (...)`). `code/tt_sam3/benchmark.py` reads the
same knob (same device open, same fused paths), so its PCC numbers gate the fused path.
Sub-knobs (`SAM3_ENC_TRACE`, `SAM3_ENC_FUSED`, `SAM3_BB_RESIDENCY`, `SAM3_BB_FUSED_RESIDUAL`,
`SAM3_BB_MATMUL`, `SAM3_BB_INPUT`, `SAM3_BB_BF16_READBACK`, `SAM3_DIT_BLOCKS`,
`SAM3_MM_BLOCKS`) and their validated defaults are documented in
`code/tt_sam3/tt/fused_config.py`; the measured numbers, kept/dropped levers and the A/B
ladder in `DEVICE_VALIDATION.md` ("Results"). Served on this p150a: forward medians 472-490 ms
fused vs 540-578 ms legacy (4×50 warm requests each over two sessions), identical instance
counts, top score 0.9729 vs 0.9727 on the demo request.

## 2. Package / serve / push (container)

Run **from this directory**: `source.extra_code[].root: code` and `--out` resolve against
the process CWD, not the YAML. Rootless Docker is in use on this box, so every tt-model /
docker command needs `source $ROOT/bin/docker-env.sh` first.

```bash
ROOT=/home/deepgadget/experiments/tt-models
source $ROOT/bin/docker-env.sh
cd $ROOT/models/sam3-p150

# offline validation (no build): must print VALID
$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest as L; m=L('tt-model.yaml', check_sources=True); p=m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"

# build (hours cold; the C++ tree rebuilds whenever code/ changes), then serve, then push
$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out $ROOT/build
$ROOT/.venv/bin/tt-model serve   $ROOT/build/sam3-p150/tt_kernel_manifest.json
python3 code/tt_sam3/server/smoke_test.py --url http://127.0.0.1:<port serve printed>
$ROOT/.venv/bin/tt-model stop    $ROOT/build/sam3-p150/tt_kernel_manifest.json
$ROOT/.venv/bin/tt-model push    $ROOT/build/sam3-p150 --publish
```

What the launcher does for this kind: installs `torch==2.11.0+cpu` (read from the tree's
`requirements-dev.txt`) + fastapi/uvicorn/pydantic/pillow + `runtime.packages`; at build
time imports `tt_sam3.server.app` as uid 1000 with no device/weights and runs every
`verify:` line (they check `sam3.model_builder`, triton/psutil/torchvision.ops, torchvision
0.26.0, numpy 1.x, the vendored BPE file and that the SAM3 tokenizer can load it); at serve
time pre-downloads `sam3.pt` + `config.json` at the pinned revision into the host HF cache
(mounted at `/hf`), then runs
`python -m uvicorn --host 0.0.0.0 --port <port> --lifespan on tt_sam3.server.app:app` with
`HF_MODEL=facebook/sam3 MESH_DEVICE=P150 TT_MESH_SHAPE=1x1` + `serve.env`
(`TT_WEIGHTS_REVISION`, `TT_METAL_VISIBLE_DEVICES=0`, `SAM3_BPE_PATH`,
`SAM3_TORCH_COMPILE=0`, `SAM3_WARMUP_RUNS=2`, `TT_FUSED=1`) and waits up to 30 min for
`Application startup complete`. `stop` sends SIGTERM and waits 120 s.

tt-cli users: `tt serve changh95/sam3-p150`, `tt model stop changh95/sam3-p150`.

Push replaces `README.md` with the generated card (everything worth keeping lives in
`card.description` / `card.quickstart` of `tt-model.yaml`) and makes `code/` on the Hub
exactly the staged allowlist (`tt_sam3`, `scripts`, `conftest.py`). `media/`, this file,
`.gitattributes` and `tt-model.yaml` survive at the repo root. The orchestrator restores
`license` / `pipeline_tag` / extra tags after the push.

## 3. The HTTP contract

`POST /predict` (JSON):

| field | type | default | meaning |
|---|---|---|---|
| `image` | base64 string | required | PNG/JPEG, any size, RGB/RGBA; squashed bilinearly to the fixed 1008x1008 the port validated, normalised mean=std=0.5 |
| `prompts` | `[str]` | required | 1..4 noun phrases (`SAM3_MAX_PROMPTS`); each is an independent query and costs one full forward |
| `threshold` | float 0..1 | 0.5 | keep instances with `sigmoid(logit) * sigmoid(presence) > threshold` (upstream `Sam3Processor` semantics) |
| `max_instances` | int 1..200 | 50 | per prompt, highest scores first |
| `mask_format` | `png` \| `rle` \| `none` | `png` | `png`: base64 8-bit 0/255 PNG at the original HxW; `rle`: COCO uncompressed RLE (`{size:[H,W], counts:[...]}`, column-major); `none`: boxes/scores only |

Response:

```json
{"width": 1920, "height": 1080, "input_size": 1008, "threshold": 0.5, "mask_format": "png",
 "results": [{"prompt": "car", "presence": 0.99, "num_instances": 3,
              "instances": [{"score": 0.978, "box_xyxy": [x0, y0, x1, y1], "mask_area": 123456,
                             "mask_png": "<base64>"}]}],
 "timing_ms": {"decode": ., "preprocess": ., "forward": ., "forward_per_prompt": ., "postprocess": ., "total": .}}
```

`box_xyxy` is in ORIGINAL image pixels (clamped to the image); masks are the 288x288
logits bilinearly upsampled to the original size and thresholded at 0 (== sigmoid > 0.5, as
upstream). Errors: 400 bad image / empty prompt / too many prompts / image side >
`SAM3_MAX_SIDE`; 422 schema violations; 503 while starting; 500 with the exception text.

`GET /health` -> `{"status": "ok"|"starting", "model": "sam3-p150", "device": {...}}`;
`GET /info` -> weights repo + revision actually loaded, checkpoint path, limits, source
repo + commit, runtime switches; `GET /v1/models` -> an OpenAI-shaped list so the tt-model
ready card and `tt-model curl` do not 404 (they are not this API).

## 4. Caveats

- **Gated weights.** `facebook/sam3` needs Meta's approval and a logged-in token on the
  host. The app loads `sam3.pt` via `hf_hub_download(revision=<sha>)` and passes
  `checkpoint_path=` to `sam3` (`load_from_HF=False`), so the pin is honoured and a
  pre-downloaded snapshot resolves from the cache without network (verified offline with
  `HF_HUB_OFFLINE=1`). Without `TT_WEIGHTS_REVISION` it resolves `main` on the Hub.
- **BPE vocabulary.** The PyPI `sam3==0.1.4` wheel does NOT ship `sam3/assets/`, so
  `build_sam3_image_model(bpe_path=None)` fails. The vocabulary (1,356,917 bytes, sha256
  `924691ac...6804a`, from upstream `facebookresearch/sam3`) is vendored at
  `code/tt_sam3/assets/bpe_simple_vocab_16e6.txt.gz` (git-LFS via `*.gz`) and resolved by
  `tt_sam3.tt.ttnn_sam3_model.resolve_bpe_path()` (env > vendored > legacy open_clip path).
- **No torch.compile in the image.** The runtime image has no C++ compiler; the decoder
  runs eager (`SAM3_TORCH_COMPILE=0`, ~3% slower than the README's compiled rows).
- **sam3 import chain.** `sam3.model_builder` imports `triton` (`sam3/model/edt.py`),
  `psutil` and `torchvision.ops` at module level; they are in `runtime.packages`
  (`triton==3.6.0` is torch 2.11's own pin; `torchvision==0.26.0+cpu` matches torch
  2.11.0+cpu). Verified: `import sam3.model_builder` succeeds on CPU-only torch 2.11.
- **Fixed shape, batch 1, text prompts only.** Every prompt re-runs the ViT (~170 ms) +
  CPU stages; there is no image-feature caching. Box/point prompts are not exposed
  (`enable_inst_interactivity=False`). Non-square images are squashed like upstream.
- **CPU-bound stages.** Neck / text encoder / decoder / seg head run in torch on the host;
  throughput depends on cores (`SAM3_NUM_THREADS`, default 17 capped at `cpu_count`).
- **Concurrency.** One chip, one globally monkey-patched torch model: every forward runs
  under a process-wide lock (post-processing runs outside it). Never run multiple uvicorn
  workers.
- **Device open.** The server opens the chip through `fused_config.device_open_kwargs`:
  `l1_small_size=0` and, for the fused default, `trace_region_size=90000000` (the encoder trace;
  32 MB was measured to suffice); `TT_FUSED=0` opens bare as the original E2E scripts did.
  `SAM3_L1_SMALL_SIZE` exists only for experiments.
- **Text cache.** The per-prompt text-feature cache is bounded (`SAM3_TEXT_CACHE_MAX`, 256
  entries, cleared when full) so a long-running server does not grow without limit.
- **Not hardware-validated yet** in this package form: the hardware phase runs
  `smoke_test.py` against the served container and compares with `media/target_1_ttnn.png`.