--- license: apache-2.0 library_name: custom tags: - beam - build-with-beam - modernbert - decision-model - candidate-scoring - synthetic-supervision --- # Scotty Scotty is a compact, 421-million-parameter decision model trained with Beam-generated data and Typed Decisions examples. It reads a situation, a question, and the supplied answer options together, then scores all options in a single forward pass. It supports boolean, categorical, and ordered decisions without generating free-form text. - Inference code: [`run_scotty.py`](https://huggingface.co/DarwinGuru/scotty/blob/main/run_scotty.py) - Paper: [`paper/scotty-paper.pdf`](https://huggingface.co/DarwinGuru/scotty/blob/main/paper/scotty-paper.pdf) - Demo: (access-controlled) ## Contact For research questions or demo access, contact Oliver Kramer at . ## How it works Scotty combines a pretrained `answerdotai/ModernBERT-large` encoder with a newly initialized two-layer Laya-format decision head. Each option receives a `[MASK]` marker. A shared scalar head scores those marker states, and softmax ranks the options. Candidates are encoded jointly rather than in separate chunks. The output values are comparative scores, not calibrated confidence. Scotty does not produce explanations or hidden reasoning. ## Training data Reflection Beam 1.0 501B-A23B generated 19,878 typed decisions across 18 task families. Beam supplied supervision, not weights. Fitting uses **11,948 Beam decisions plus 4,320 Typed decisions** from `LocalLLaMA/typed-decisions`. The earlier 13,292/4,800 counts described source pools before reserving 1,344/480 audit decisions. The released weights match the audited finalist and its fitting configuration; see [provenance verification](results/controlled-20261110/provenance_verification.json). Every effective batch is split 50:50 between Beam-generated and Typed examples, with uniform sampling with replacement inside each source. This does not guarantee every fitting item is seen. The final model uses seed 0, 849 optimizer updates, an effective batch size of 64, BF16 autocast, and full encoder fine-tuning. The Beam dataset-generation API calls stayed within the $25 participant credit budget. This excludes local training and hardware costs. ### How Beam was used This Build with Beam project uses Reflection Beam 1.0 501B-A23B as the teacher that generated typed decision examples. Scotty does not contain Beam weights: it learns from the generated situations, questions, options, and target choices. ## Controlled mixture experiment (10 November 2026) Twelve fixed-budget runs compare source ratio and within-source drawing rule independently: four conditions, three paired seeds, 849 updates and 54,336 presentations per run. These are separate from the unchanged released Scotty checkpoint. | Mixture | Drawing rule | Typed development | Beam balanced development | Typed test | |---|---|---:|---:|---:| | Proportional | Cycle | 76.92 ± 1.51 | 92.15 ± 0.52 | 75.10 ± 0.80 | | 50:50 | Cycle | 79.50 ± 0.74 | 92.10 ± 0.87 | 77.15 ± 0.72 | | Proportional | Replacement | 78.14 ± 0.63 | 91.88 ± 0.33 | 75.55 ± 0.88 | | 50:50 | Replacement | 78.86 ± 0.32 | 91.71 ± 0.37 | 77.47 ± 0.50 | Percent, mean ± sample SD across three seeds. Equal mixing improves retrospective Typed test accuracy by 2.05 points under cycling (conditional paired-case 95% interval 1.03–3.10) and 1.92 points under replacement (0.97–2.92). Intervals resample 400 cases with 10,000 draws and condition on the three observed models; they do not estimate seed-population uncertainty. All development, audit and official test sets had been observed historically. No checkpoint or recipe was selected using these retrospective scores. ### Laya context Under the common evaluator, released Scotty scores **76.80% (1,536/2,000)**, the **local Laya specialist reproduction 74.35% (1,487/2,000)**, and general Laya 36.35%. Scotty's descriptive advantage over the local reproduction is 2.45 points (paired-case interval 0.65–4.20). This is not the released Laya specialist: its [model card](https://huggingface.co/convaiinnovations/laya-typed-decisions/blob/main/README.md) reports **76.6%**, and an [independent rerun](https://arxiv.org/abs/2609.33843) reports **76.7%** on the official test. Those externally reported scores were not produced by our evaluator; no paired comparison with that released specialist is established. Differences in training data, initialization and budget also preclude an architecture-isolated claim. [Evidence, split manifest and evaluator](https://github.com/evolution-strategies/scotty/tree/main/results/controlled-20261110). The same sanitized bundle is included in [results/controlled-20261110](results/controlled-20261110/README.md). ## Files ```text model.safetensors scotty_config.json run_scotty.py requirements.txt paper/scotty-paper.pdf paper/scotty-reader-source.zip encoder/config.json tokenizer/ results/controlled-20261110/ ``` `scotty_config.json` records the encoder, decision-head shape, token budgets, precision, and release provenance. The included `run_scotty.py` adapter gives Laya 0.3.28 the legacy filename it expects only inside a temporary compatibility view; the published checkpoint does not use an RL-labelled manifest. Scotty was trained with supervised learning. `model.safetensors` SHA-256: ```text e79b758aea590bcc4504e6b622c206080a7d1fedc49d572d3a185321b8e2eeda ``` ## Download and run ```bash python3 -m pip install huggingface_hub python3 -c 'from huggingface_hub import snapshot_download; snapshot_download("DarwinGuru/scotty", local_dir="scotty")' cd scotty python3 -m pip install -r requirements.txt python3 run_scotty.py \ --checkpoint . \ --verify-only python3 run_scotty.py \ --checkpoint . \ --device cpu \ --state "The shipment is sealed, labelled, and the manifest is signed." \ --instruction "Is this shipment ready for dispatch?" \ --option false \ --option true ``` Scotty is a custom Laya-format checkpoint, not a drop-in Transformers text- classification pipeline. ## Limitations - Both supervision sources are synthetic; teacher errors can be learned. - Three seeds are a small sample. - The source audit is not an independent external test. - Scores require calibration before confidence-based automation. - No real-world deployment study or broad throughput benchmark is reported. ## License Original Scotty checkpoint weights are Apache-2.0. That grant does not automatically license datasets, the paper, or third-party material. This model card, the checkpoint configuration, and the release files record the relevant revisions, attribution, and license scope. See [`LICENSE_SCOPE.md`](https://huggingface.co/DarwinGuru/scotty/blob/main/LICENSE_SCOPE.md) for the files in this release. Key upstreams are `answerdotai/ModernBERT-large` revision `45bb4654a4d5aaff24dd11d4781fa46d39bf8c13`, Laya `0.3.28` (training checkout `eba3b0367643498e478ed02e870fa53737da5061`), and `LocalLLaMA/typed-decisions` revision `468b1461d59e01d6404e1cf37431da962acd43a0`. Each retains its own attribution and terms.