File size: 8,511 Bytes
6b5f702
 
7de56e1
 
 
 
 
 
 
 
 
 
 
f8398fb
 
7de56e1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6b5f702
7de56e1
 
 
 
 
 
 
f8398fb
 
 
 
 
 
 
 
 
 
 
 
7de56e1
 
f8398fb
 
 
 
7de56e1
 
 
 
 
f8398fb
 
 
 
 
 
 
 
 
 
7de56e1
f8398fb
 
 
 
 
 
7de56e1
 
 
f8398fb
 
 
 
7de56e1
 
 
 
f8398fb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7de56e1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f8398fb
 
7de56e1
f8398fb
 
 
 
7de56e1
 
f8398fb
 
7de56e1
f8398fb
7de56e1
f8398fb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
7de56e1
 
 
 
f8398fb
7de56e1
f8398fb
7de56e1
 
 
 
f8398fb
 
7de56e1
 
 
 
f8398fb
 
 
 
 
 
 
7de56e1
 
 
f8398fb
 
7de56e1
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
---

license: mit
language:
  - en
tags:
  - ndna
  - neural-dna
  - genome
  - sparse
  - gpt2
  - topology
  - pruning
  - connectivity
  - neural-architecture-search
  - developmental-encoding
datasets:
  - openwebtext
metrics:
  - perplexity
model-index:
  - name: ndna-gpt2-small
    results:
      - task:
          type: text-generation
          name: Language Modeling
        dataset:
          name: WikiText-103
          type: wikitext
        metrics:
          - type: perplexity
            value: 36.0
            name: Perplexity
      - task:
          type: text-generation
          name: Language Modeling
        dataset:
          name: Penn Treebank
          type: ptb_text_only
        metrics:
          - type: perplexity
            value: 59.4
            name: Perplexity
      - task:
          type: text-generation
          name: Language Modeling
        dataset:
          name: LAMBADA
          type: lambada
        metrics:
          - type: perplexity
            value: 22.2
            name: Perplexity
---


# NDNA GPT-2 Small

A GPT-2 Small (124M params) model where a **354-parameter genome** controls 35.4 million connections. The genome decides which attention output projections and feed-forward layers are connected, permanently disabling one-third of masked connections.

**99,970:1 compression ratio.** 354 learned parameters specify the wiring for 35.4M connections.

## How It Works

Neural DNA (NDNA) encodes network connectivity in a compact genome. Instead of storing one mask bit per connection, the genome stores a developmental program: 8 cell types with 8-dimensional affinity vectors and a compatibility matrix. Source and target type embeddings are compared to produce connection probabilities, which are thresholded to binary masks using a straight-through estimator. A metabolic cost term encourages sparsity, forcing the genome to be selective about which connections survive.

The genome and model weights are trained jointly from scratch on OpenWebText. Temperature annealing (1.0 to 10.0) smoothly transitions from soft to hard binary masks, locking the topology by the end of training.

**What the genome controls:**
- **Masked:** W_O (768x768, bias=False) and FF1 (3072x768) per layer = 2.95M connections/layer, 35.4M total across 12 layers

- **Not masked:** Q, K, V projections, FF2, LayerNorm, residual connections, LM head



The genome controls 28.4% of the model's total parameters. The remaining 71.6% train freely.



## Results



![Training Loss](figures/fig1_training_loss.png)



*Validation loss over 52,700 iterations. Best checkpoint: 3.04 at iteration 49,500.*



| Benchmark | NDNA GPT-2 | GPT-2 Small | Result |

|-----------|-----------|-------------|--------|

| WikiText-103 PPL | **36.0** | 37.5 | Beats GPT-2 |

| Penn Treebank PPL | **59.4** | 65.9 | Beats GPT-2 |

| LAMBADA PPL | **22.2** | 35.1 | Beats GPT-2 |

| LAMBADA ACC | 30.8% | 46.0% | 67% of GPT-2 |

| HellaSwag ACC | 28.7% | 31.2% | 92% of GPT-2 |

| CBT-CN ACC | 82.7% | 87.7% | 94% of GPT-2 |

| CBT-NE ACC | 74.5% | 83.4% | 89% of GPT-2 |

| enwiki8 BPB | 1.39 | 1.16 | |

| text8 BPC | 1.27 | 1.17 | |



Beats GPT-2 on 3 of 9 benchmarks (all perplexity metrics). The model produces better-calibrated probability distributions but less peaked predictions, leading to lower perplexity but lower exact-match accuracy.



### Supplementary Results (Language Model Evaluation Harness)



| Benchmark | Metric | NDNA GPT-2 |

|-----------|--------|-----------|

| ARC-Easy | ACC | 41.8% |

| PIQA | ACC | 60.1% |

| Winogrande | ACC | 52.8% |

| HellaSwag | ACC (norm) | 28.8% |



## Learned Topology



![Final Topology](figures/fig3_final_topology.png)



*Final per-layer hard density at temperature 10.0. Sharp boundary between sparse (L1-L4) and dense (L5-L12) zones.*



| Layer | 1 | 2 | 3 | 4 | 5-12 |

|-------|------|------|------|-----|------|

| Hard Density | 98.2% | 45.2% | 20.4% | 7.7% | 100% |



Layers 5-12 are fully connected. Layers 1-4 are progressively pruned in a strict monotonic gradient. The genome treats the network as having two distinct zones. One-third of all masked connections (11.8M out of 35.4M) are permanently disabled.



## Training Dynamics



![Density Heatmap](figures/fig2_density_heatmap.png)



*Per-layer hard density over training. Layer 1 is pruned at iter 800, then re-activates at iter 26,000.*



The genome goes through four distinct phases:



1. **Over-activation (iter 0-200):** Genome activates 83% of connections. Everything is on.

2. **Aggressive pruning (iter 200-800):** Layers 1-4 are pruned to 0% hard density. The genome discovers the model can function with only 8 of 12 layers carrying signal through masked projections.

3. **Stable learning (iter 800-25K):** Topology is fixed. Validation loss drops from 7.99 to 3.18. The network learns language with a sparse topology.

4. **Layer 1 resurrection (iter 25K-50K):** Layer 1 re-activates from 0% to 98.2% hard density. The genome changed its mind. Layers 2-4 remain pruned.



The layer 1 resurrection is the most unexpected finding. After 25,000 iterations of being completely disabled, the genome reverses its decision and reconnects layer 1. This is a discrete topological event, not a smooth interpolation.



## Usage



```python

import torch

from genome.model import Genome, GrownGPT2



# Load checkpoint

ckpt = torch.load("genome_ckpt.pt", map_location="cpu")



# Initialize

genome = Genome(n_types=8, type_dim=8, n_bands=14)
model = GrownGPT2(genome)

# Load weights (strip torch.compile prefix)
model_state = {k.replace("_orig_mod.", ""): v for k, v in ckpt["model"].items()}

genome_state = {k.replace("_orig_mod.", ""): v for k, v in ckpt["genome"].items()}
genome.load_state_dict(genome_state)

model.load_state_dict(model_state, strict=False)

# Enable hard binary masks for evaluation
model.hard_masks = True

model.eval()

```



## Training Details



- **Dataset:** OpenWebText (open reproduction of WebText)

- **Hardware:** Single NVIDIA A100 80GB GPU

- **Iterations:** 52,700 (best checkpoint at 49,500)

- **Tokens:** 25.9 billion (491,520 tokens/iter = 12 seqs x 1024 tokens x 40 grad accum)

- **Weight optimizer:** AdamW (lr=6e-4, beta1=0.9, beta2=0.95, cosine decay, warmup 2000, weight decay 0.1)

- **Genome optimizer:** Adam (lr=0.01, constant, no decay)

- **Temperature:** 1.0 to 10.0 over 50K iterations (linear anneal)

- **Sparsity weight:** 0.005

- **Precision:** bfloat16 with torch.compile

- **Gradient clipping:** 1.0

- **Random seed:** 42



### Genome Configuration



| Component | Shape | Parameters |

|-----------|-------|-----------|

| Affinity matrix A | 8x8 | 64 |

| Compatibility matrix C | 8x8 | 64 |

| Connection scale | scalar | 1 |

| Depth penalty | scalar | 1 |

| Band type base | 14x8 | 112 |

| Band type gradient | 14x8 | 112 |

| **Total** | | **354** |



8 cell types, 8-dimensional affinity vectors, 14 bands (1 embedding + 12 transformer layers + 1 output).



### Compression Ratio Progression



| Experiment | Genome Params | Connections | Compression |

|------------|--------------|-------------|-------------|

| MNIST MLP | 226 | 174K | 770:1 |

| CIFAR-10 MLP | 226 | 1.7M | 7,553:1 |

| IMDB Transformer | 258 | 2.2M | 8,384:1 |

| **GPT-2 Small** | **354** | **35.4M** | **99,970:1** |



## Links



- **Paper 1:** [Neural DNA: A Compact Genome for Growing Network Architecture](https://doi.org/10.5281/zenodo.19248389)

- **Paper 2:** [Scaling Neural DNA to GPT-2: 354 Parameters Wire a Language Model](https://zenodo.org/records/19390927)

- **Code:** [github.com/tejassudsfp/ndna](https://github.com/tejassudsfp/ndna)

- **Interactive Visualization:** [ndna.tejassuds.com](https://ndna.tejassuds.com)



## Author



Tejas Parthasarathi Sudarshan

Independent Researcher, Chennai, India

[[email protected]](mailto:[email protected]) | [tejassuds.com](https://tejassuds.com) | [LinkedIn](https://www.linkedin.com/in/tejassuds/)



## Citation



```bibtex

@article{sudarshan2026ndna,

  title={Neural DNA: A Compact Genome for Growing Network Architecture},

  author={Sudarshan, Tejas Parthasarathi},

  year={2026},

  doi={10.5281/zenodo.19248389}

}



@article{sudarshan2026ndna_gpt2,
  title={Scaling Neural DNA to GPT-2: 354 Parameters Wire a Language Model},
  author={Sudarshan, Tejas Parthasarathi},
  year={2026},
  doi={10.5281/zenodo.19390927}
}
```