Datdanboi25 Claude Opus 4.8 commited on
Commit
9bd52d1
·
1 Parent(s): 4de6eb2

Add GPT-S2-5M model: weights, config, tokenizer, modeling code

Browse files

~5.4M-param T-X model with RoPE, RMSNorm, SwiGLU, exclusive grouped-query
attention, and the XSA refresh gate. fp32 safetensors weights, custom 4K
tokenizer, and trust_remote_code modeling/conversion code.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ *.png filter=lfs diff=lfs merge=lfs -text
.gitignore ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ __pycache__
2
+ data_cache
3
+ log
4
+ log_adam
5
+ decay_cp
6
+ decay_cp_adam
7
+ train_gpt-s3-5m.py
8
+ convert_to_hf.py
9
+ verify_hf.py
10
+ $null
11
+ .claude
AxiomicBanner.png ADDED

Git LFS Details

  • SHA256: 75c38db1d76736a243905a29ba6e85eaf65e14037cd78c4f391bc146fe36fe4e
  • Pointer size: 130 Bytes
  • Size of remote file: 88 kB
README.md CHANGED
@@ -1,3 +1,149 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ language:
4
+ - en
5
+ library_name: transformers
6
+ tags:
7
+ - language-model
8
+ - transformer
9
+ - rope
10
+ - swiglu
11
+ - gqa
12
+ - xsa
13
+ - xsa-refresh
14
+ - custom-architecture
15
+ - custom-tokenizer
16
+ pipeline_tag: text-generation
17
  ---
18
+
19
+ ![axiomic banner](AxiomicBanner.png)
20
+
21
+ # GPT-S2-5M
22
+
23
+ GPT-S2-5M is a ~5.4M parameter model in the GPT-S small-model family: a custom
24
+ 4K tokenizer, 9 layers, and the T-X architecture with the **XSA refresh gate**,
25
+ trained from scratch.
26
+
27
+ It combines RoPE + RMSNorm + SwiGLU + exclusive grouped-query attention with an
28
+ **XSA refresh gate** on two interior layers: a gated, causal depthwise-conv path
29
+ that re-injects the original token embedding back into the residual stream,
30
+ conditioned on the (detached) attention output.
31
+
32
+ ## Architecture
33
+
34
+ | Component | Details |
35
+ |---|---|
36
+ | Position encoding | RoPE, theta = 2,500 |
37
+ | Normalization | RMSNorm (eps 1e-6) |
38
+ | Feed-forward | SwiGLU |
39
+ | Attention | Exclusive grouped-query attention (XSA), 6 query heads / 2 KV heads |
40
+ | Refresh | XSA refresh gate on layers 5 and 9 (0-indexed 4, 8), kernel 9 |
41
+ | Embeddings | Weight tied |
42
+ | Context length | 512 tokens |
43
+ | Parameters | 5,384,258 |
44
+
45
+ ### Config
46
+
47
+ ```text
48
+ vocab_size = 4,096
49
+ hidden_size = 192
50
+ num_layers = 9
51
+ num_heads = 6
52
+ num_kv_heads = 2
53
+ head_dim = 32
54
+ intermediate = 672
55
+ block_size = 512
56
+ rope_theta = 2,500
57
+ inject_layers = [4, 8]
58
+ refresh_kernel = 9
59
+ ```
60
+
61
+ ### The XSA refresh gate
62
+
63
+ On the injection layers, after the attention residual add, the block applies:
64
+
65
+ ```text
66
+ a = RMSNorm(attn_out.detach()) # read attention as a signal
67
+ e = RMSNorm(e0) # original token embedding
68
+ gate = gate_proj(a) + causal_depthwise_conv(a) # kernel-9, depthwise
69
+ value = value_proj(e)
70
+ z = RMSNorm(out_proj(SiLU(gate) * value))
71
+ x = x + alpha * z # alpha is a learned scalar
72
+ ```
73
+
74
+ `e0` is the token embedding from the input layer, re-injected at every gate.
75
+ The depthwise conv is strictly causal, so the gate is compatible with KV-cache
76
+ generation (the conv state is carried alongside the attention cache).
77
+
78
+ ## Benchmarks
79
+
80
+ Zero-shot, evaluated in bf16 with an internal harness modeled on
81
+ EleutherAI/lm-eval-harness; normalized accuracy where available. Scores below
82
+ are from this checkpoint (step 281,250).
83
+
84
+ | Hellaswag | ARC (avg) | PIQA | Arithmark-2 | Average |
85
+ | --------- | --------- | ---- | ----------- | ------- |
86
+ | 27.57% | 28.39% | 57.56% | 28.04% | 35.39% |
87
+
88
+ ## Training
89
+
90
+ | Hyperparameter | Value |
91
+ |---|---:|
92
+ | Optimizers | AdamW (embeddings / LM head / norms / conv) + Muon (2D hidden weights) |
93
+ | Adam betas | 0.9 / 0.95 |
94
+ | Adam peak LR | 2.5e-3 |
95
+ | Muon peak LR | 0.03 |
96
+ | Muon momentum | 0.95 |
97
+ | Weight decay | 0.01 |
98
+ | Minimum LR | 0 |
99
+ | LR schedule | Warmup-stable-decay |
100
+ | Warmup steps | 1,500 |
101
+ | Decay start | 85% of training |
102
+ | Training tokens | ~73.7B (step 281,250 × 262,144) |
103
+ | Total batch size | 262,144 tokens |
104
+ | Microbatch | 128 × 512 tokens |
105
+ | Gradient accumulation steps | 4 |
106
+ | Gradient clipping | 1.0 |
107
+ | Precision | bfloat16 autocast |
108
+
109
+ Trained on a mixture of filtered web text (DCLM) and synthetic instructional
110
+ "finephrase" data (FAQ / math / table / tutorial), with a small benchmark-style
111
+ suite folded in.
112
+
113
+ ## Usage
114
+
115
+ ```python
116
+ from transformers import AutoModelForCausalLM, AutoTokenizer
117
+ import torch
118
+
119
+ model_name = "AxiomicLabs/GPT-S2-5M"
120
+
121
+ tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
122
+ model = AutoModelForCausalLM.from_pretrained(
123
+ model_name,
124
+ trust_remote_code=True,
125
+ torch_dtype=torch.bfloat16,
126
+ device_map="auto",
127
+ )
128
+
129
+ prompt = "The future of AI is"
130
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
131
+
132
+ with torch.inference_mode():
133
+ output = model.generate(
134
+ **inputs,
135
+ max_new_tokens=120,
136
+ do_sample=True,
137
+ temperature=0.8,
138
+ top_p=0.95,
139
+ repetition_penalty=1.1,
140
+ no_repeat_ngram_size=4,
141
+ )
142
+
143
+ print(tokenizer.decode(output[0], skip_special_tokens=True))
144
+ ```
145
+
146
+ ## Limitations
147
+
148
+ This is a small base language model. It is not instruction tuned, has limited
149
+ factual capacity, and uses a 512-token context window.
__init__.py ADDED
File without changes
config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "GPTS3ForCausalLM"
4
+ ],
5
+ "attention_type": "exclusive_grouped_query",
6
+ "auto_map": {
7
+ "AutoConfig": "configuration_gpts3.GPTS3Config",
8
+ "AutoModelForCausalLM": "modeling_gpts3.GPTS3ForCausalLM"
9
+ },
10
+ "bias": false,
11
+ "dtype": "float32",
12
+ "embedding_scale": false,
13
+ "head_dim": 32,
14
+ "hidden_act": "silu",
15
+ "hidden_size": 192,
16
+ "inject_layers": [
17
+ 4,
18
+ 8
19
+ ],
20
+ "intermediate_size": 672,
21
+ "max_position_embeddings": 512,
22
+ "model_type": "gpts3",
23
+ "num_attention_heads": 6,
24
+ "num_hidden_layers": 9,
25
+ "num_key_value_heads": 2,
26
+ "refresh_kernel": 9,
27
+ "rms_norm_eps": 1e-06,
28
+ "rope_theta": 2500.0,
29
+ "tie_word_embeddings": true,
30
+ "transformers_version": "5.5.3",
31
+ "vocab_size": 4096,
32
+ "xsa_projection": true
33
+ }
configuration_gpts3.py ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """GPT-S3 model configuration."""
2
+ from transformers import PretrainedConfig
3
+
4
+
5
+ class GPTS3Config(PretrainedConfig):
6
+ model_type = "gpts3"
7
+
8
+ def __init__(
9
+ self,
10
+ vocab_size=4096,
11
+ hidden_size=192,
12
+ num_hidden_layers=9,
13
+ num_attention_heads=6,
14
+ num_key_value_heads=2,
15
+ head_dim=32,
16
+ intermediate_size=672,
17
+ max_position_embeddings=512,
18
+ rope_theta=2500.0,
19
+ rms_norm_eps=1e-6,
20
+ tie_word_embeddings=True,
21
+ xsa_projection=True,
22
+ inject_layers=(4, 8),
23
+ refresh_kernel=9,
24
+ **kwargs,
25
+ ):
26
+ self.vocab_size = vocab_size
27
+ self.hidden_size = hidden_size
28
+ self.num_hidden_layers = num_hidden_layers
29
+ self.num_attention_heads = num_attention_heads
30
+ self.num_key_value_heads = num_key_value_heads
31
+ self.head_dim = head_dim
32
+ self.intermediate_size = intermediate_size
33
+ self.max_position_embeddings = max_position_embeddings
34
+ self.rope_theta = rope_theta
35
+ self.rms_norm_eps = rms_norm_eps
36
+ self.xsa_projection = xsa_projection
37
+ # Layers (0-indexed) that carry the XSA refresh gate.
38
+ self.inject_layers = list(inject_layers)
39
+ self.refresh_kernel = refresh_kernel
40
+ super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)
generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 2,
6
+ "transformers_version": "5.5.3"
7
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a2598a520db405403ba86839307d16a12ccb20faed141c2f9f8b395c916b7e2e
3
+ size 21547352
modeling_gpts3.py ADDED
@@ -0,0 +1,313 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """GPT-S3 model implementation for Hugging Face transformers.
2
+
3
+ Architecture (T-X3 with XSA refresh): RoPE + RMSNorm + SwiGLU + exclusive
4
+ grouped-query attention (XSA projection), plus an XSA *refresh gate* that
5
+ re-injects the original token embedding through a gated depthwise-causal-conv
6
+ path on a subset of layers (``config.inject_layers``).
7
+ """
8
+ from typing import Optional
9
+
10
+ import torch
11
+ import torch.nn as nn
12
+ from torch.nn import functional as F
13
+ from transformers import PreTrainedModel
14
+ from transformers.cache_utils import DynamicCache
15
+ from transformers.generation.utils import GenerationMixin
16
+ from transformers.modeling_outputs import CausalLMOutputWithPast
17
+
18
+ try:
19
+ from .configuration_gpts3 import GPTS3Config
20
+ except ImportError: # allow flat import when not loaded as a package
21
+ from configuration_gpts3 import GPTS3Config
22
+
23
+
24
+ class RMSNorm(nn.Module):
25
+ def __init__(self, dim, eps=1e-6):
26
+ super().__init__()
27
+ self.eps = eps
28
+ self.weight = nn.Parameter(torch.ones(dim))
29
+
30
+ def forward(self, x):
31
+ rms = torch.rsqrt(x.float().pow(2).mean(-1, keepdim=True) + self.eps)
32
+ return (x.float() * rms).type_as(x) * self.weight
33
+
34
+
35
+ def precompute_rope_cos_sin(head_dim, seq_len, theta=2500.0):
36
+ freqs = 1.0 / (theta ** (torch.arange(0, head_dim, 2, dtype=torch.float32) / head_dim))
37
+ t = torch.arange(seq_len, dtype=torch.float32)
38
+ freqs = torch.outer(t, freqs)
39
+ return freqs.cos(), freqs.sin()
40
+
41
+
42
+ def apply_rotary_emb(q, k, rope_cos, rope_sin):
43
+ # rope_cos/rope_sin: [T, head_dim/2]
44
+ cos = rope_cos.unsqueeze(0).unsqueeze(0)
45
+ sin = rope_sin.unsqueeze(0).unsqueeze(0)
46
+
47
+ q_float = q.float().reshape(*q.shape[:-1], -1, 2)
48
+ k_float = k.float().reshape(*k.shape[:-1], -1, 2)
49
+
50
+ q_even, q_odd = q_float.unbind(-1)
51
+ k_even, k_odd = k_float.unbind(-1)
52
+
53
+ q_out = torch.stack((q_even * cos - q_odd * sin, q_even * sin + q_odd * cos), dim=-1).flatten(-2)
54
+ k_out = torch.stack((k_even * cos - k_odd * sin, k_even * sin + k_odd * cos), dim=-1).flatten(-2)
55
+ return q_out.type_as(q), k_out.type_as(k)
56
+
57
+
58
+ class GPTS3Attention(nn.Module):
59
+ def __init__(self, config, layer_idx):
60
+ super().__init__()
61
+ self.layer_idx = layer_idx
62
+ self.n_head = config.num_attention_heads
63
+ self.n_kv_heads = config.num_key_value_heads
64
+ self.head_dim = config.head_dim
65
+ self.n_rep = self.n_head // self.n_kv_heads
66
+ self.xsa_projection = config.xsa_projection
67
+
68
+ self.q_proj = nn.Linear(config.hidden_size, self.n_head * self.head_dim, bias=False)
69
+ self.k_proj = nn.Linear(config.hidden_size, self.n_kv_heads * self.head_dim, bias=False)
70
+ self.v_proj = nn.Linear(config.hidden_size, self.n_kv_heads * self.head_dim, bias=False)
71
+ self.o_proj = nn.Linear(self.n_head * self.head_dim, config.hidden_size, bias=False)
72
+
73
+ def forward(self, x, rope_cos, rope_sin, past_key_value=None, use_cache=False, attention_mask=None):
74
+ B, T, _ = x.size()
75
+ q = self.q_proj(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)
76
+ k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
77
+ v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
78
+ v_current = v # current-token values (pre-cache), used by the XSA projection
79
+
80
+ q, k = apply_rotary_emb(q, k, rope_cos, rope_sin)
81
+
82
+ if past_key_value is not None:
83
+ k, v = past_key_value.update(k, v, self.layer_idx)
84
+
85
+ S = k.size(2)
86
+ k = k.unsqueeze(2).expand(B, self.n_kv_heads, self.n_rep, S, self.head_dim).reshape(B, self.n_head, S, self.head_dim)
87
+ v = v.unsqueeze(2).expand(B, self.n_kv_heads, self.n_rep, S, self.head_dim).reshape(B, self.n_head, S, self.head_dim)
88
+
89
+ is_causal = past_key_value is None or past_key_value.get_seq_length(self.layer_idx) == T
90
+ attn_mask = None
91
+ if attention_mask is not None:
92
+ key_pad = attention_mask.to(torch.bool)[:, None, None, :]
93
+ if is_causal and T > 1:
94
+ causal = torch.ones(T, S, dtype=torch.bool, device=x.device).tril(diagonal=S - T)
95
+ attn_mask = key_pad & causal[None, None, :, :]
96
+ else:
97
+ attn_mask = key_pad.expand(B, 1, T, S)
98
+ is_causal = False
99
+
100
+ y = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal)
101
+
102
+ # XSA projection: remove the component of each head's output along its
103
+ # group's current-token value vector. y_proj = y - (<y,v>/<v,v>) v.
104
+ if self.xsa_projection:
105
+ if v.size(2) == T:
106
+ v_base = v
107
+ else:
108
+ v_base = v_current.unsqueeze(2).expand(B, self.n_kv_heads, self.n_rep, T, self.head_dim).reshape(B, self.n_head, T, self.head_dim)
109
+ denom = v_base.pow(2).sum(dim=-1, keepdim=True).clamp_min(1e-6)
110
+ y = y - (y * v_base).sum(dim=-1, keepdim=True) / denom * v_base
111
+
112
+ y = y.transpose(1, 2).contiguous().view(B, T, self.n_head * self.head_dim)
113
+ return self.o_proj(y)
114
+
115
+
116
+ class GPTS3SwiGLUMLP(nn.Module):
117
+ def __init__(self, config):
118
+ super().__init__()
119
+ self.w_gate = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
120
+ self.w_up = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
121
+ self.w_down = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
122
+
123
+ def forward(self, x):
124
+ return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
125
+
126
+
127
+ class XSARefreshGate(nn.Module):
128
+ """Re-injects the original token embedding (e0) into the residual stream,
129
+ gated by a causal depthwise conv over the (detached) attention output."""
130
+
131
+ def __init__(self, d_model, kernel=9):
132
+ super().__init__()
133
+ self.na = RMSNorm(d_model)
134
+ self.ne = RMSNorm(d_model)
135
+
136
+ self.gate_proj = nn.Linear(d_model, d_model, bias=False)
137
+ self.gate_conv = nn.Conv1d(d_model, d_model, kernel, groups=d_model, bias=False, padding=kernel - 1)
138
+
139
+ self.value_proj = nn.Linear(d_model, d_model, bias=False)
140
+ self.out_proj = nn.Linear(d_model, d_model, bias=False)
141
+ self.nz = RMSNorm(d_model)
142
+
143
+ self.alpha = nn.Parameter(torch.tensor(0.0))
144
+ self.kernel = kernel
145
+
146
+ def forward(self, h, attn_out, e0, conv_state=None, layer_idx=None):
147
+ a = self.na(attn_out.detach()) # gate reads attention as a signal only
148
+ e = self.ne(e0) # grad flows into tied wte/lm_head
149
+
150
+ k = self.kernel
151
+ B, T, D = a.shape
152
+ if conv_state is not None:
153
+ prev = conv_state.get(layer_idx)
154
+ if prev is None or prev.size(0) != B:
155
+ prev = a.new_zeros(B, k - 1, D)
156
+ a_ext = torch.cat([prev, a], dim=1) # [B, (k-1)+T, D]
157
+ conv_state[layer_idx] = a_ext[:, -(k - 1):, :].detach()
158
+ else:
159
+ a_ext = F.pad(a, (0, 0, k - 1, 0)) # left-pad time with zeros
160
+ # Manual left-pad + padding=0 conv reproduces the training-time causal
161
+ # depthwise conv (padding=k-1 then truncated to T) for any T.
162
+ c = F.conv1d(a_ext.transpose(1, 2), self.gate_conv.weight, bias=None, padding=0, groups=D)
163
+ c = c.transpose(1, 2) # [B, T, D]
164
+
165
+ gate = self.gate_proj(a) + c
166
+ value = self.value_proj(e)
167
+ z = F.silu(gate) * value
168
+ z = self.nz(self.out_proj(z))
169
+ return h + self.alpha * z
170
+
171
+
172
+ class GPTS3Block(nn.Module):
173
+ def __init__(self, config, layer_idx):
174
+ super().__init__()
175
+ self.layer_idx = layer_idx
176
+ self.ln_1 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
177
+ self.attn = GPTS3Attention(config, layer_idx)
178
+ self.refresh = (
179
+ XSARefreshGate(config.hidden_size, kernel=config.refresh_kernel)
180
+ if layer_idx in config.inject_layers
181
+ else None
182
+ )
183
+ self.ln_2 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
184
+ self.mlp = GPTS3SwiGLUMLP(config)
185
+
186
+ def forward(self, x, e0, rope_cos, rope_sin, past_key_value=None, use_cache=False,
187
+ attention_mask=None, conv_state=None):
188
+ attn_out = self.attn(self.ln_1(x), rope_cos, rope_sin, past_key_value, use_cache, attention_mask=attention_mask)
189
+ x = x + attn_out
190
+ if self.refresh is not None:
191
+ x = self.refresh(x, attn_out, e0, conv_state=conv_state, layer_idx=self.layer_idx)
192
+ x = x + self.mlp(self.ln_2(x))
193
+ return x
194
+
195
+
196
+ class GPTS3PreTrainedModel(PreTrainedModel):
197
+ config_class = GPTS3Config
198
+ base_model_prefix = "transformer"
199
+ supports_gradient_checkpointing = False
200
+
201
+ def _init_weights(self, module):
202
+ std = self.config.hidden_size ** -0.5
203
+ if isinstance(module, nn.Linear):
204
+ torch.nn.init.normal_(module.weight, mean=0.0, std=std)
205
+ elif isinstance(module, nn.Embedding):
206
+ torch.nn.init.normal_(module.weight, mean=0.0, std=std)
207
+ elif isinstance(module, nn.Conv1d):
208
+ torch.nn.init.normal_(module.weight, mean=0.0, std=std)
209
+
210
+
211
+ class GPTS3ForCausalLM(GPTS3PreTrainedModel, GenerationMixin):
212
+ _tied_weights_keys = {"lm_head.weight": "transformer.wte.weight"}
213
+
214
+ def __init__(self, config):
215
+ super().__init__(config)
216
+ self.config = config
217
+ self.transformer = nn.ModuleDict(dict(
218
+ wte=nn.Embedding(config.vocab_size, config.hidden_size),
219
+ h=nn.ModuleList([GPTS3Block(config, i) for i in range(config.num_hidden_layers)]),
220
+ ln_f=RMSNorm(config.hidden_size, eps=config.rms_norm_eps),
221
+ ))
222
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
223
+ if config.tie_word_embeddings:
224
+ self.lm_head.weight = self.transformer["wte"].weight
225
+ self._rope_cache = None # (cos, sin) cached on device up to some length
226
+ self.post_init()
227
+
228
+ def get_input_embeddings(self):
229
+ return self.transformer["wte"]
230
+
231
+ def set_input_embeddings(self, value):
232
+ self.transformer["wte"] = value
233
+
234
+ def get_output_embeddings(self):
235
+ return self.lm_head
236
+
237
+ def set_output_embeddings(self, new_embeddings):
238
+ self.lm_head = new_embeddings
239
+
240
+ def prepare_inputs_for_generation(self, input_ids, past_key_values=None, attention_mask=None, **kwargs):
241
+ if past_key_values is not None and past_key_values.get_seq_length() > 0:
242
+ input_ids = input_ids[:, -1:]
243
+ return {
244
+ "input_ids": input_ids,
245
+ "attention_mask": attention_mask,
246
+ "past_key_values": past_key_values,
247
+ "use_cache": True,
248
+ }
249
+
250
+ def _get_rope(self, seq_len, device):
251
+ cache = self._rope_cache
252
+ if cache is None or cache[0].device != device or cache[0].size(0) < seq_len:
253
+ cos, sin = precompute_rope_cos_sin(self.config.head_dim, seq_len, self.config.rope_theta)
254
+ cache = (cos.to(device), sin.to(device))
255
+ self._rope_cache = cache
256
+ return cache[0][:seq_len], cache[1][:seq_len]
257
+
258
+ def forward(
259
+ self,
260
+ input_ids,
261
+ attention_mask=None,
262
+ labels=None,
263
+ past_key_values: Optional[DynamicCache] = None,
264
+ use_cache=False,
265
+ **kwargs,
266
+ ):
267
+ B, T = input_ids.size()
268
+ conv_state = None
269
+ if use_cache:
270
+ if past_key_values is None:
271
+ past_key_values = DynamicCache()
272
+ # Conv-state cache for the refresh gate lives alongside the KV cache,
273
+ # so it shares the same lifetime as a single generation.
274
+ if not hasattr(past_key_values, "_refresh_conv_state"):
275
+ past_key_values._refresh_conv_state = {}
276
+ conv_state = past_key_values._refresh_conv_state
277
+
278
+ past_len = past_key_values.get_seq_length() if past_key_values is not None else 0
279
+ x = self.transformer["wte"](input_ids)
280
+ e0 = x # original token embeddings, re-injected at every refresh layer
281
+
282
+ cos, sin = self._get_rope(past_len + T, input_ids.device)
283
+ cos, sin = cos[past_len:], sin[past_len:]
284
+
285
+ for block in self.transformer["h"]:
286
+ x = block(
287
+ x, e0, cos, sin,
288
+ past_key_values if use_cache else None,
289
+ use_cache,
290
+ attention_mask=attention_mask,
291
+ conv_state=conv_state,
292
+ )
293
+
294
+ x = self.transformer["ln_f"](x)
295
+ logits = self.lm_head(x)
296
+
297
+ loss = None
298
+ if labels is not None:
299
+ shift_logits = logits[..., :-1, :].contiguous()
300
+ shift_labels = labels[..., 1:].contiguous()
301
+ loss = F.cross_entropy(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
302
+
303
+ return CausalLMOutputWithPast(
304
+ loss=loss,
305
+ logits=logits,
306
+ past_key_values=past_key_values if use_cache else None,
307
+ )
308
+
309
+
310
+ # Backward-compatible aliases for configs/checkpoints created under earlier
311
+ # family names.
312
+ GPTX3ForCausalLM = GPTS3ForCausalLM
313
+ GPTS3ForCausalLM = GPTS3ForCausalLM
special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<bos>",
3
+ "eos_token": "<eos>",
4
+ "unk_token": "<unk>",
5
+ "pad_token": "<eos>"
6
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": true,
3
+ "added_tokens_decoder": {
4
+ "0": {
5
+ "content": "<unk>",
6
+ "lstrip": false,
7
+ "normalized": false,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "1": {
13
+ "content": "<bos>",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "2": {
21
+ "content": "<eos>",
22
+ "lstrip": false,
23
+ "normalized": false,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ }
28
+ },
29
+ "bos_token": "<bos>",
30
+ "eos_token": "<eos>",
31
+ "unk_token": "<unk>",
32
+ "pad_token": "<eos>",
33
+ "clean_up_tokenization_spaces": false,
34
+ "model_max_length": 512,
35
+ "tokenizer_class": "PreTrainedTokenizerFast",
36
+ "vocab_size": 4096
37
+ }
usage.py ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import AutoModelForCausalLM, AutoTokenizer
2
+ import torch
3
+
4
+ model_name = "AxiomicLabs/GPT-S2-5M"
5
+
6
+ tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
7
+ model = AutoModelForCausalLM.from_pretrained(
8
+ model_name,
9
+ trust_remote_code=True,
10
+ torch_dtype=torch.bfloat16,
11
+ device_map="auto",
12
+ )
13
+
14
+ prompt = "The future of AI is"
15
+ inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
16
+
17
+ with torch.inference_mode():
18
+ output = model.generate(
19
+ **inputs,
20
+ max_new_tokens=120,
21
+ do_sample=True,
22
+ temperature=0.8,
23
+ top_p=0.95,
24
+ repetition_penalty=1.1,
25
+ no_repeat_ngram_size=4,
26
+ )
27
+
28
+ text = tokenizer.decode(output[0], skip_special_tokens=True)
29
+ print(text)