# Resumable fx2 QAT worker (swagged-out-bot)

`qat256_worker.py` is the QAT + cooldown + gate + export worker behind our q256 run. It's shared because it
survives kills: every `--every` steps (default 100) rank 0 saves model + Muon + AdamW + step to
`<out>/ckpt_<phase>.pt`. A relaunch with the same arguments prints `RESUMED at step N` and skips the batches
already done (the loader's article order is fixed, seed 0). A finished phase leaves `<phase>.tch`, so it is never repeated.

Run it on one GPU with torchrun:

    torchrun --standalone --nproc_per_node=1 qat256_worker.py --data <train priors dir> --eval <3 MB gate dir> \
        --init <fine-tune .tch> --out <dir> --embed15 1 --accum 16 [--compile 1]

- `--accum 16` on one GPU gives the same 16 x 2^16 tokens per step as 4 GPUs x 4.
- It expects the upstream repo at /root/s (edit the two `/root/s` lines).
- `--compile 1` gives 1.95x, but only with the integer bf16 scale-rounding patch the worker applies to
  pysrc/quantization.py. It runs a compiled-vs-eager interlock before training: <= 1e-4 of weights may differ, by
  at most 1 level. Unpatched compile is silently wrong (0/111 tensors bit-identical).
- Memory: the priors are a 23 GB float16 file (56.78 M tokens x 410 B), opened with mmap. On a 29 GB host the
  page cache competes with other jobs, so the checkpoint interval is the real protection.
- Continuation (warm restart) of a finished run: `--init <qat.tch> --opt-from <ckpt_qat.pt> --tag cont
  --qat-peak 3 --init-steps 0 --qat-epochs N --cd-epochs 0|16 --control-eval 1`.

Tested on Modal L40S with 4-GPU DDP and 1 GPU. The smoke test kills the run mid-QAT on purpose and checks that the
restart prints RESUMED and finishes. Our q256 (QAT 256 + cooldown 16) ran on it.