--- license: mit tags: - transformer - cryptography - ablation-study - from-scratch --- # Cipher-to-Plaintext Transformers — Architectural Ablation Encoder–decoder transformers built from fundamental PyTorch operations (no `nn.Transformer`, no `nn.MultiheadAttention`) for ANLP Assignment 1. The task: recover English plaintext from a byte-wise repeating-key XOR cipher with an 8-byte key, one byte (8 bits) per plaintext character. The key itself is a property of the course dataset and is not published here. ## Configurations Each differs from the C1 baseline by exactly one component. | | Positional | Attention | Norm | Tokenization | |---|---|---|---|---| | **C1** | Sinusoidal | Multi-head | LayerNorm | byte-level BPE | | **C2** | **RoPE** | Multi-head | LayerNorm | byte-level BPE | | **C3** | Sinusoidal | **Grouped-query** (2 kv heads) | LayerNorm | byte-level BPE | | **C4** | Sinusoidal | Multi-head | **RMSNorm** | byte-level BPE | | **C5** | Sinusoidal | Multi-head | LayerNorm | **BLT, token-free** | Shared hyperparameters: `d_model` 256, 8 heads, 6 layers, `d_ff` 1024, dropout 0.1, AdamW at 3e-4, batch 32, 20 epochs, 64-character chunks. ## Results Greedy decoding on a held-out test split of 4,371 chunks. | | seq acc | char acc | bit acc | Levenshtein | params | s/epoch | peak GB | |---|---:|---:|---:|---:|---:|---:|---:| | **C1** | 96.71% | 99.8928% | 99.9630% | 0.064 | 11,316,480 | 56.7 | 0.95 | | **C2** | 97.16% | 99.3812% | 99.7809% | 0.069 | 11,316,480 | 58.7 | 0.94 | | **C3** | 97.21% | 99.8899% | 99.9623% | 0.065 | 9,547,008 | 52.4 | 0.92 | | **C4** | 97.46% | 99.9178% | 99.9723% | 0.051 | 11,308,288 | 55.1 | 0.88 | | **C5** | 99.50% | 99.9904% | 99.9962% | 0.006 | 14,532,608 | 50.1 | 0.93 | Measured noise floor (C1, three seeds): ±0.0175pp character accuracy. Differences smaller than that are not interpreted. ## Files - `C1_BitPairTokenizer_gelu_best.pt` (45.9 MB) - `C1_BytePairTokenizer_gelu_best.pt` (45.9 MB) - `C1_BytePairTokenizer_relu_best.pt` (45.9 MB) - `C2_BitPairTokenizer_gelu_best.pt` (45.4 MB) - `C2_BytePairTokenizer_gelu_best.pt` (45.4 MB) - `C2_BytePairTokenizer_relu_best.pt` (45.4 MB) - `C3_BitPairTokenizer_gelu_best.pt` (38.8 MB) - `C3_BytePairTokenizer_gelu_best.pt` (38.8 MB) - `C3_BytePairTokenizer_relu_best.pt` (38.8 MB) - `C4_BitPairTokenizer_gelu_best.pt` (45.8 MB) - `C4_BytePairTokenizer_gelu_best.pt` (45.8 MB) - `C4_BytePairTokenizer_relu_best.pt` (45.8 MB) - `C5E_ByteTokenizer_gelu_best.pt` (60.3 MB) - `C5_ByteTokenizer_gelu_best.pt` (59.3 MB) - `C5_ByteTokenizer_relu_best.pt` (59.3 MB) - `histories/C1_BitPairTokenizer_gelu_history.json` (0.0 MB) - `histories/C1_BytePairTokenizer_gelu_history.json` (0.0 MB) - `histories/C1_BytePairTokenizer_relu_history.json` (0.0 MB) - `histories/C2_BitPairTokenizer_gelu_history.json` (0.0 MB) - `histories/C2_BytePairTokenizer_gelu_history.json` (0.0 MB) - `histories/C2_BytePairTokenizer_relu_history.json` (0.0 MB) - `histories/C3_BitPairTokenizer_gelu_history.json` (0.0 MB) - `histories/C3_BytePairTokenizer_gelu_history.json` (0.0 MB) - `histories/C3_BytePairTokenizer_relu_history.json` (0.0 MB) - `histories/C4_BitPairTokenizer_gelu_history.json` (0.0 MB) - `histories/C4_BytePairTokenizer_gelu_history.json` (0.0 MB) - `histories/C4_BytePairTokenizer_relu_history.json` (0.0 MB) - `histories/C5E_ByteTokenizer_gelu_history.json` (0.0 MB) - `histories/C5_ByteTokenizer_gelu_history.json` (0.0 MB) - `histories/C5_ByteTokenizer_relu_history.json` (0.0 MB) - `figures/attention_C1.png` (0.0 MB) - `figures/attention_C2.png` (0.0 MB) - `figures/attention_C3.png` (0.0 MB) - `figures/attention_C4.png` (0.0 MB) - `figures/attention_C5.png` (0.0 MB) - `figures/bars_memory.png` (0.1 MB) - `figures/bars_seq_acc.png` (0.1 MB) - `figures/bars_speed.png` (0.1 MB) - `figures/bitbpe_attention_C1.png` (0.1 MB) - `figures/bitbpe_attention_C2.png` (0.0 MB) - `figures/bitbpe_attention_C3.png` (0.0 MB) - `figures/bitbpe_attention_C4.png` (0.1 MB) - `figures/bitbpe_bars_memory.png` (0.0 MB) - `figures/bitbpe_bars_seq_acc.png` (0.0 MB) - `figures/bitbpe_bars_speed.png` (0.0 MB) - `figures/bitbpe_curves_val_acc.png` (0.1 MB) - `figures/bitbpe_curves_val_loss.png` (0.1 MB) - `figures/c5e_attention_C5E.png` (0.0 MB) - `figures/c5e_bars_memory.png` (0.0 MB) - `figures/c5e_bars_seq_acc.png` (0.0 MB) - `figures/c5e_bars_speed.png` (0.0 MB) - `figures/c5e_curves_val_acc.png` (0.1 MB) - `figures/c5e_curves_val_loss.png` (0.0 MB) - `figures/curves_val_acc.png` (0.1 MB) - `figures/curves_val_loss.png` (0.2 MB) - `bpe_vocab1024.json` (0.0 MB) - `results.json` (0.0 MB) - `heldout_result.json` (0.0 MB) ## Loading a checkpoint These are raw `state_dict` files for an architecture implemented from scratch, so they need this project's model code: ```python from train import build, TOKENIZERS import torch tokenizer = TOKENIZERS["bpe"]() # "byte" for C5 model = build("C1", tokenizer, activation="gelu") model.load_state_dict(torch.load("C1_BytePairTokenizer_gelu_best.pt")) model.eval() ``` `bpe_vocab1024.json` is the trained tokenizer and is required for C1–C4; C5 uses raw bytes and needs no vocabulary. ## Notes - Reported numbers use greedy decoding, as the assignment specifies. Accuracy logged during training is teacher-forced and is not comparable. - BLEU and ROUGE are reported for the tokenized configurations only; they saturate near 0.99 on this task and do not discriminate between configurations. - A held-out combination experiment shows the model memorises a phase-independent byte→character table rather than inferring the cipher. Withholding 20 of the 424 (character, phase) combinations drops accuracy on them from 99.96% to 23.8% — and every combination whose key byte is unique across the key scores **exactly 0%**. The residual comes entirely from one key byte that repeats at two phases, so those combinations produce cipher bytes the model had already seen elsewhere. No structural generalisation occurs.