Text Generation
Transformers
Safetensors
English
gpts3
language-model
transformer
rope
swiglu
gqa
xsa
xsa-refresh
custom-architecture
custom-tokenizer
custom_code
Instructions to use AxiomicLabs/GPT-S2-5M with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AxiomicLabs/GPT-S2-5M with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="AxiomicLabs/GPT-S2-5M", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("AxiomicLabs/GPT-S2-5M", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use AxiomicLabs/GPT-S2-5M with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AxiomicLabs/GPT-S2-5M" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AxiomicLabs/GPT-S2-5M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/AxiomicLabs/GPT-S2-5M
- SGLang
How to use AxiomicLabs/GPT-S2-5M with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AxiomicLabs/GPT-S2-5M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AxiomicLabs/GPT-S2-5M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AxiomicLabs/GPT-S2-5M" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AxiomicLabs/GPT-S2-5M", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use AxiomicLabs/GPT-S2-5M with Docker Model Runner:
docker model run hf.co/AxiomicLabs/GPT-S2-5M
Commit ·
9bd52d1
1
Parent(s): 4de6eb2
Add GPT-S2-5M model: weights, config, tokenizer, modeling code
Browse files~5.4M-param T-X model with RoPE, RMSNorm, SwiGLU, exclusive grouped-query
attention, and the XSA refresh gate. fp32 safetensors weights, custom 4K
tokenizer, and trust_remote_code modeling/conversion code.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- .gitattributes +1 -0
- .gitignore +11 -0
- AxiomicBanner.png +3 -0
- README.md +146 -0
- __init__.py +0 -0
- config.json +33 -0
- configuration_gpts3.py +40 -0
- generation_config.json +7 -0
- model.safetensors +3 -0
- modeling_gpts3.py +313 -0
- special_tokens_map.json +6 -0
- tokenizer.json +0 -0
- tokenizer_config.json +37 -0
- usage.py +29 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
*.png filter=lfs diff=lfs merge=lfs -text
|
.gitignore
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
__pycache__
|
| 2 |
+
data_cache
|
| 3 |
+
log
|
| 4 |
+
log_adam
|
| 5 |
+
decay_cp
|
| 6 |
+
decay_cp_adam
|
| 7 |
+
train_gpt-s3-5m.py
|
| 8 |
+
convert_to_hf.py
|
| 9 |
+
verify_hf.py
|
| 10 |
+
$null
|
| 11 |
+
.claude
|
AxiomicBanner.png
ADDED
|
Git LFS Details
|
README.md
CHANGED
|
@@ -1,3 +1,149 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
library_name: transformers
|
| 6 |
+
tags:
|
| 7 |
+
- language-model
|
| 8 |
+
- transformer
|
| 9 |
+
- rope
|
| 10 |
+
- swiglu
|
| 11 |
+
- gqa
|
| 12 |
+
- xsa
|
| 13 |
+
- xsa-refresh
|
| 14 |
+
- custom-architecture
|
| 15 |
+
- custom-tokenizer
|
| 16 |
+
pipeline_tag: text-generation
|
| 17 |
---
|
| 18 |
+
|
| 19 |
+

|
| 20 |
+
|
| 21 |
+
# GPT-S2-5M
|
| 22 |
+
|
| 23 |
+
GPT-S2-5M is a ~5.4M parameter model in the GPT-S small-model family: a custom
|
| 24 |
+
4K tokenizer, 9 layers, and the T-X architecture with the **XSA refresh gate**,
|
| 25 |
+
trained from scratch.
|
| 26 |
+
|
| 27 |
+
It combines RoPE + RMSNorm + SwiGLU + exclusive grouped-query attention with an
|
| 28 |
+
**XSA refresh gate** on two interior layers: a gated, causal depthwise-conv path
|
| 29 |
+
that re-injects the original token embedding back into the residual stream,
|
| 30 |
+
conditioned on the (detached) attention output.
|
| 31 |
+
|
| 32 |
+
## Architecture
|
| 33 |
+
|
| 34 |
+
| Component | Details |
|
| 35 |
+
|---|---|
|
| 36 |
+
| Position encoding | RoPE, theta = 2,500 |
|
| 37 |
+
| Normalization | RMSNorm (eps 1e-6) |
|
| 38 |
+
| Feed-forward | SwiGLU |
|
| 39 |
+
| Attention | Exclusive grouped-query attention (XSA), 6 query heads / 2 KV heads |
|
| 40 |
+
| Refresh | XSA refresh gate on layers 5 and 9 (0-indexed 4, 8), kernel 9 |
|
| 41 |
+
| Embeddings | Weight tied |
|
| 42 |
+
| Context length | 512 tokens |
|
| 43 |
+
| Parameters | 5,384,258 |
|
| 44 |
+
|
| 45 |
+
### Config
|
| 46 |
+
|
| 47 |
+
```text
|
| 48 |
+
vocab_size = 4,096
|
| 49 |
+
hidden_size = 192
|
| 50 |
+
num_layers = 9
|
| 51 |
+
num_heads = 6
|
| 52 |
+
num_kv_heads = 2
|
| 53 |
+
head_dim = 32
|
| 54 |
+
intermediate = 672
|
| 55 |
+
block_size = 512
|
| 56 |
+
rope_theta = 2,500
|
| 57 |
+
inject_layers = [4, 8]
|
| 58 |
+
refresh_kernel = 9
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
### The XSA refresh gate
|
| 62 |
+
|
| 63 |
+
On the injection layers, after the attention residual add, the block applies:
|
| 64 |
+
|
| 65 |
+
```text
|
| 66 |
+
a = RMSNorm(attn_out.detach()) # read attention as a signal
|
| 67 |
+
e = RMSNorm(e0) # original token embedding
|
| 68 |
+
gate = gate_proj(a) + causal_depthwise_conv(a) # kernel-9, depthwise
|
| 69 |
+
value = value_proj(e)
|
| 70 |
+
z = RMSNorm(out_proj(SiLU(gate) * value))
|
| 71 |
+
x = x + alpha * z # alpha is a learned scalar
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
+
`e0` is the token embedding from the input layer, re-injected at every gate.
|
| 75 |
+
The depthwise conv is strictly causal, so the gate is compatible with KV-cache
|
| 76 |
+
generation (the conv state is carried alongside the attention cache).
|
| 77 |
+
|
| 78 |
+
## Benchmarks
|
| 79 |
+
|
| 80 |
+
Zero-shot, evaluated in bf16 with an internal harness modeled on
|
| 81 |
+
EleutherAI/lm-eval-harness; normalized accuracy where available. Scores below
|
| 82 |
+
are from this checkpoint (step 281,250).
|
| 83 |
+
|
| 84 |
+
| Hellaswag | ARC (avg) | PIQA | Arithmark-2 | Average |
|
| 85 |
+
| --------- | --------- | ---- | ----------- | ------- |
|
| 86 |
+
| 27.57% | 28.39% | 57.56% | 28.04% | 35.39% |
|
| 87 |
+
|
| 88 |
+
## Training
|
| 89 |
+
|
| 90 |
+
| Hyperparameter | Value |
|
| 91 |
+
|---|---:|
|
| 92 |
+
| Optimizers | AdamW (embeddings / LM head / norms / conv) + Muon (2D hidden weights) |
|
| 93 |
+
| Adam betas | 0.9 / 0.95 |
|
| 94 |
+
| Adam peak LR | 2.5e-3 |
|
| 95 |
+
| Muon peak LR | 0.03 |
|
| 96 |
+
| Muon momentum | 0.95 |
|
| 97 |
+
| Weight decay | 0.01 |
|
| 98 |
+
| Minimum LR | 0 |
|
| 99 |
+
| LR schedule | Warmup-stable-decay |
|
| 100 |
+
| Warmup steps | 1,500 |
|
| 101 |
+
| Decay start | 85% of training |
|
| 102 |
+
| Training tokens | ~73.7B (step 281,250 × 262,144) |
|
| 103 |
+
| Total batch size | 262,144 tokens |
|
| 104 |
+
| Microbatch | 128 × 512 tokens |
|
| 105 |
+
| Gradient accumulation steps | 4 |
|
| 106 |
+
| Gradient clipping | 1.0 |
|
| 107 |
+
| Precision | bfloat16 autocast |
|
| 108 |
+
|
| 109 |
+
Trained on a mixture of filtered web text (DCLM) and synthetic instructional
|
| 110 |
+
"finephrase" data (FAQ / math / table / tutorial), with a small benchmark-style
|
| 111 |
+
suite folded in.
|
| 112 |
+
|
| 113 |
+
## Usage
|
| 114 |
+
|
| 115 |
+
```python
|
| 116 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 117 |
+
import torch
|
| 118 |
+
|
| 119 |
+
model_name = "AxiomicLabs/GPT-S2-5M"
|
| 120 |
+
|
| 121 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
| 122 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 123 |
+
model_name,
|
| 124 |
+
trust_remote_code=True,
|
| 125 |
+
torch_dtype=torch.bfloat16,
|
| 126 |
+
device_map="auto",
|
| 127 |
+
)
|
| 128 |
+
|
| 129 |
+
prompt = "The future of AI is"
|
| 130 |
+
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
| 131 |
+
|
| 132 |
+
with torch.inference_mode():
|
| 133 |
+
output = model.generate(
|
| 134 |
+
**inputs,
|
| 135 |
+
max_new_tokens=120,
|
| 136 |
+
do_sample=True,
|
| 137 |
+
temperature=0.8,
|
| 138 |
+
top_p=0.95,
|
| 139 |
+
repetition_penalty=1.1,
|
| 140 |
+
no_repeat_ngram_size=4,
|
| 141 |
+
)
|
| 142 |
+
|
| 143 |
+
print(tokenizer.decode(output[0], skip_special_tokens=True))
|
| 144 |
+
```
|
| 145 |
+
|
| 146 |
+
## Limitations
|
| 147 |
+
|
| 148 |
+
This is a small base language model. It is not instruction tuned, has limited
|
| 149 |
+
factual capacity, and uses a 512-token context window.
|
__init__.py
ADDED
|
File without changes
|
config.json
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"GPTS3ForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attention_type": "exclusive_grouped_query",
|
| 6 |
+
"auto_map": {
|
| 7 |
+
"AutoConfig": "configuration_gpts3.GPTS3Config",
|
| 8 |
+
"AutoModelForCausalLM": "modeling_gpts3.GPTS3ForCausalLM"
|
| 9 |
+
},
|
| 10 |
+
"bias": false,
|
| 11 |
+
"dtype": "float32",
|
| 12 |
+
"embedding_scale": false,
|
| 13 |
+
"head_dim": 32,
|
| 14 |
+
"hidden_act": "silu",
|
| 15 |
+
"hidden_size": 192,
|
| 16 |
+
"inject_layers": [
|
| 17 |
+
4,
|
| 18 |
+
8
|
| 19 |
+
],
|
| 20 |
+
"intermediate_size": 672,
|
| 21 |
+
"max_position_embeddings": 512,
|
| 22 |
+
"model_type": "gpts3",
|
| 23 |
+
"num_attention_heads": 6,
|
| 24 |
+
"num_hidden_layers": 9,
|
| 25 |
+
"num_key_value_heads": 2,
|
| 26 |
+
"refresh_kernel": 9,
|
| 27 |
+
"rms_norm_eps": 1e-06,
|
| 28 |
+
"rope_theta": 2500.0,
|
| 29 |
+
"tie_word_embeddings": true,
|
| 30 |
+
"transformers_version": "5.5.3",
|
| 31 |
+
"vocab_size": 4096,
|
| 32 |
+
"xsa_projection": true
|
| 33 |
+
}
|
configuration_gpts3.py
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""GPT-S3 model configuration."""
|
| 2 |
+
from transformers import PretrainedConfig
|
| 3 |
+
|
| 4 |
+
|
| 5 |
+
class GPTS3Config(PretrainedConfig):
|
| 6 |
+
model_type = "gpts3"
|
| 7 |
+
|
| 8 |
+
def __init__(
|
| 9 |
+
self,
|
| 10 |
+
vocab_size=4096,
|
| 11 |
+
hidden_size=192,
|
| 12 |
+
num_hidden_layers=9,
|
| 13 |
+
num_attention_heads=6,
|
| 14 |
+
num_key_value_heads=2,
|
| 15 |
+
head_dim=32,
|
| 16 |
+
intermediate_size=672,
|
| 17 |
+
max_position_embeddings=512,
|
| 18 |
+
rope_theta=2500.0,
|
| 19 |
+
rms_norm_eps=1e-6,
|
| 20 |
+
tie_word_embeddings=True,
|
| 21 |
+
xsa_projection=True,
|
| 22 |
+
inject_layers=(4, 8),
|
| 23 |
+
refresh_kernel=9,
|
| 24 |
+
**kwargs,
|
| 25 |
+
):
|
| 26 |
+
self.vocab_size = vocab_size
|
| 27 |
+
self.hidden_size = hidden_size
|
| 28 |
+
self.num_hidden_layers = num_hidden_layers
|
| 29 |
+
self.num_attention_heads = num_attention_heads
|
| 30 |
+
self.num_key_value_heads = num_key_value_heads
|
| 31 |
+
self.head_dim = head_dim
|
| 32 |
+
self.intermediate_size = intermediate_size
|
| 33 |
+
self.max_position_embeddings = max_position_embeddings
|
| 34 |
+
self.rope_theta = rope_theta
|
| 35 |
+
self.rms_norm_eps = rms_norm_eps
|
| 36 |
+
self.xsa_projection = xsa_projection
|
| 37 |
+
# Layers (0-indexed) that carry the XSA refresh gate.
|
| 38 |
+
self.inject_layers = list(inject_layers)
|
| 39 |
+
self.refresh_kernel = refresh_kernel
|
| 40 |
+
super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)
|
generation_config.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"_from_model_config": true,
|
| 3 |
+
"bos_token_id": 1,
|
| 4 |
+
"eos_token_id": 2,
|
| 5 |
+
"pad_token_id": 2,
|
| 6 |
+
"transformers_version": "5.5.3"
|
| 7 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a2598a520db405403ba86839307d16a12ccb20faed141c2f9f8b395c916b7e2e
|
| 3 |
+
size 21547352
|
modeling_gpts3.py
ADDED
|
@@ -0,0 +1,313 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""GPT-S3 model implementation for Hugging Face transformers.
|
| 2 |
+
|
| 3 |
+
Architecture (T-X3 with XSA refresh): RoPE + RMSNorm + SwiGLU + exclusive
|
| 4 |
+
grouped-query attention (XSA projection), plus an XSA *refresh gate* that
|
| 5 |
+
re-injects the original token embedding through a gated depthwise-causal-conv
|
| 6 |
+
path on a subset of layers (``config.inject_layers``).
|
| 7 |
+
"""
|
| 8 |
+
from typing import Optional
|
| 9 |
+
|
| 10 |
+
import torch
|
| 11 |
+
import torch.nn as nn
|
| 12 |
+
from torch.nn import functional as F
|
| 13 |
+
from transformers import PreTrainedModel
|
| 14 |
+
from transformers.cache_utils import DynamicCache
|
| 15 |
+
from transformers.generation.utils import GenerationMixin
|
| 16 |
+
from transformers.modeling_outputs import CausalLMOutputWithPast
|
| 17 |
+
|
| 18 |
+
try:
|
| 19 |
+
from .configuration_gpts3 import GPTS3Config
|
| 20 |
+
except ImportError: # allow flat import when not loaded as a package
|
| 21 |
+
from configuration_gpts3 import GPTS3Config
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
class RMSNorm(nn.Module):
|
| 25 |
+
def __init__(self, dim, eps=1e-6):
|
| 26 |
+
super().__init__()
|
| 27 |
+
self.eps = eps
|
| 28 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 29 |
+
|
| 30 |
+
def forward(self, x):
|
| 31 |
+
rms = torch.rsqrt(x.float().pow(2).mean(-1, keepdim=True) + self.eps)
|
| 32 |
+
return (x.float() * rms).type_as(x) * self.weight
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def precompute_rope_cos_sin(head_dim, seq_len, theta=2500.0):
|
| 36 |
+
freqs = 1.0 / (theta ** (torch.arange(0, head_dim, 2, dtype=torch.float32) / head_dim))
|
| 37 |
+
t = torch.arange(seq_len, dtype=torch.float32)
|
| 38 |
+
freqs = torch.outer(t, freqs)
|
| 39 |
+
return freqs.cos(), freqs.sin()
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def apply_rotary_emb(q, k, rope_cos, rope_sin):
|
| 43 |
+
# rope_cos/rope_sin: [T, head_dim/2]
|
| 44 |
+
cos = rope_cos.unsqueeze(0).unsqueeze(0)
|
| 45 |
+
sin = rope_sin.unsqueeze(0).unsqueeze(0)
|
| 46 |
+
|
| 47 |
+
q_float = q.float().reshape(*q.shape[:-1], -1, 2)
|
| 48 |
+
k_float = k.float().reshape(*k.shape[:-1], -1, 2)
|
| 49 |
+
|
| 50 |
+
q_even, q_odd = q_float.unbind(-1)
|
| 51 |
+
k_even, k_odd = k_float.unbind(-1)
|
| 52 |
+
|
| 53 |
+
q_out = torch.stack((q_even * cos - q_odd * sin, q_even * sin + q_odd * cos), dim=-1).flatten(-2)
|
| 54 |
+
k_out = torch.stack((k_even * cos - k_odd * sin, k_even * sin + k_odd * cos), dim=-1).flatten(-2)
|
| 55 |
+
return q_out.type_as(q), k_out.type_as(k)
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
class GPTS3Attention(nn.Module):
|
| 59 |
+
def __init__(self, config, layer_idx):
|
| 60 |
+
super().__init__()
|
| 61 |
+
self.layer_idx = layer_idx
|
| 62 |
+
self.n_head = config.num_attention_heads
|
| 63 |
+
self.n_kv_heads = config.num_key_value_heads
|
| 64 |
+
self.head_dim = config.head_dim
|
| 65 |
+
self.n_rep = self.n_head // self.n_kv_heads
|
| 66 |
+
self.xsa_projection = config.xsa_projection
|
| 67 |
+
|
| 68 |
+
self.q_proj = nn.Linear(config.hidden_size, self.n_head * self.head_dim, bias=False)
|
| 69 |
+
self.k_proj = nn.Linear(config.hidden_size, self.n_kv_heads * self.head_dim, bias=False)
|
| 70 |
+
self.v_proj = nn.Linear(config.hidden_size, self.n_kv_heads * self.head_dim, bias=False)
|
| 71 |
+
self.o_proj = nn.Linear(self.n_head * self.head_dim, config.hidden_size, bias=False)
|
| 72 |
+
|
| 73 |
+
def forward(self, x, rope_cos, rope_sin, past_key_value=None, use_cache=False, attention_mask=None):
|
| 74 |
+
B, T, _ = x.size()
|
| 75 |
+
q = self.q_proj(x).view(B, T, self.n_head, self.head_dim).transpose(1, 2)
|
| 76 |
+
k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 77 |
+
v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
|
| 78 |
+
v_current = v # current-token values (pre-cache), used by the XSA projection
|
| 79 |
+
|
| 80 |
+
q, k = apply_rotary_emb(q, k, rope_cos, rope_sin)
|
| 81 |
+
|
| 82 |
+
if past_key_value is not None:
|
| 83 |
+
k, v = past_key_value.update(k, v, self.layer_idx)
|
| 84 |
+
|
| 85 |
+
S = k.size(2)
|
| 86 |
+
k = k.unsqueeze(2).expand(B, self.n_kv_heads, self.n_rep, S, self.head_dim).reshape(B, self.n_head, S, self.head_dim)
|
| 87 |
+
v = v.unsqueeze(2).expand(B, self.n_kv_heads, self.n_rep, S, self.head_dim).reshape(B, self.n_head, S, self.head_dim)
|
| 88 |
+
|
| 89 |
+
is_causal = past_key_value is None or past_key_value.get_seq_length(self.layer_idx) == T
|
| 90 |
+
attn_mask = None
|
| 91 |
+
if attention_mask is not None:
|
| 92 |
+
key_pad = attention_mask.to(torch.bool)[:, None, None, :]
|
| 93 |
+
if is_causal and T > 1:
|
| 94 |
+
causal = torch.ones(T, S, dtype=torch.bool, device=x.device).tril(diagonal=S - T)
|
| 95 |
+
attn_mask = key_pad & causal[None, None, :, :]
|
| 96 |
+
else:
|
| 97 |
+
attn_mask = key_pad.expand(B, 1, T, S)
|
| 98 |
+
is_causal = False
|
| 99 |
+
|
| 100 |
+
y = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, is_causal=is_causal)
|
| 101 |
+
|
| 102 |
+
# XSA projection: remove the component of each head's output along its
|
| 103 |
+
# group's current-token value vector. y_proj = y - (<y,v>/<v,v>) v.
|
| 104 |
+
if self.xsa_projection:
|
| 105 |
+
if v.size(2) == T:
|
| 106 |
+
v_base = v
|
| 107 |
+
else:
|
| 108 |
+
v_base = v_current.unsqueeze(2).expand(B, self.n_kv_heads, self.n_rep, T, self.head_dim).reshape(B, self.n_head, T, self.head_dim)
|
| 109 |
+
denom = v_base.pow(2).sum(dim=-1, keepdim=True).clamp_min(1e-6)
|
| 110 |
+
y = y - (y * v_base).sum(dim=-1, keepdim=True) / denom * v_base
|
| 111 |
+
|
| 112 |
+
y = y.transpose(1, 2).contiguous().view(B, T, self.n_head * self.head_dim)
|
| 113 |
+
return self.o_proj(y)
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
class GPTS3SwiGLUMLP(nn.Module):
|
| 117 |
+
def __init__(self, config):
|
| 118 |
+
super().__init__()
|
| 119 |
+
self.w_gate = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
|
| 120 |
+
self.w_up = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
|
| 121 |
+
self.w_down = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
|
| 122 |
+
|
| 123 |
+
def forward(self, x):
|
| 124 |
+
return self.w_down(F.silu(self.w_gate(x)) * self.w_up(x))
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
class XSARefreshGate(nn.Module):
|
| 128 |
+
"""Re-injects the original token embedding (e0) into the residual stream,
|
| 129 |
+
gated by a causal depthwise conv over the (detached) attention output."""
|
| 130 |
+
|
| 131 |
+
def __init__(self, d_model, kernel=9):
|
| 132 |
+
super().__init__()
|
| 133 |
+
self.na = RMSNorm(d_model)
|
| 134 |
+
self.ne = RMSNorm(d_model)
|
| 135 |
+
|
| 136 |
+
self.gate_proj = nn.Linear(d_model, d_model, bias=False)
|
| 137 |
+
self.gate_conv = nn.Conv1d(d_model, d_model, kernel, groups=d_model, bias=False, padding=kernel - 1)
|
| 138 |
+
|
| 139 |
+
self.value_proj = nn.Linear(d_model, d_model, bias=False)
|
| 140 |
+
self.out_proj = nn.Linear(d_model, d_model, bias=False)
|
| 141 |
+
self.nz = RMSNorm(d_model)
|
| 142 |
+
|
| 143 |
+
self.alpha = nn.Parameter(torch.tensor(0.0))
|
| 144 |
+
self.kernel = kernel
|
| 145 |
+
|
| 146 |
+
def forward(self, h, attn_out, e0, conv_state=None, layer_idx=None):
|
| 147 |
+
a = self.na(attn_out.detach()) # gate reads attention as a signal only
|
| 148 |
+
e = self.ne(e0) # grad flows into tied wte/lm_head
|
| 149 |
+
|
| 150 |
+
k = self.kernel
|
| 151 |
+
B, T, D = a.shape
|
| 152 |
+
if conv_state is not None:
|
| 153 |
+
prev = conv_state.get(layer_idx)
|
| 154 |
+
if prev is None or prev.size(0) != B:
|
| 155 |
+
prev = a.new_zeros(B, k - 1, D)
|
| 156 |
+
a_ext = torch.cat([prev, a], dim=1) # [B, (k-1)+T, D]
|
| 157 |
+
conv_state[layer_idx] = a_ext[:, -(k - 1):, :].detach()
|
| 158 |
+
else:
|
| 159 |
+
a_ext = F.pad(a, (0, 0, k - 1, 0)) # left-pad time with zeros
|
| 160 |
+
# Manual left-pad + padding=0 conv reproduces the training-time causal
|
| 161 |
+
# depthwise conv (padding=k-1 then truncated to T) for any T.
|
| 162 |
+
c = F.conv1d(a_ext.transpose(1, 2), self.gate_conv.weight, bias=None, padding=0, groups=D)
|
| 163 |
+
c = c.transpose(1, 2) # [B, T, D]
|
| 164 |
+
|
| 165 |
+
gate = self.gate_proj(a) + c
|
| 166 |
+
value = self.value_proj(e)
|
| 167 |
+
z = F.silu(gate) * value
|
| 168 |
+
z = self.nz(self.out_proj(z))
|
| 169 |
+
return h + self.alpha * z
|
| 170 |
+
|
| 171 |
+
|
| 172 |
+
class GPTS3Block(nn.Module):
|
| 173 |
+
def __init__(self, config, layer_idx):
|
| 174 |
+
super().__init__()
|
| 175 |
+
self.layer_idx = layer_idx
|
| 176 |
+
self.ln_1 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
| 177 |
+
self.attn = GPTS3Attention(config, layer_idx)
|
| 178 |
+
self.refresh = (
|
| 179 |
+
XSARefreshGate(config.hidden_size, kernel=config.refresh_kernel)
|
| 180 |
+
if layer_idx in config.inject_layers
|
| 181 |
+
else None
|
| 182 |
+
)
|
| 183 |
+
self.ln_2 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
|
| 184 |
+
self.mlp = GPTS3SwiGLUMLP(config)
|
| 185 |
+
|
| 186 |
+
def forward(self, x, e0, rope_cos, rope_sin, past_key_value=None, use_cache=False,
|
| 187 |
+
attention_mask=None, conv_state=None):
|
| 188 |
+
attn_out = self.attn(self.ln_1(x), rope_cos, rope_sin, past_key_value, use_cache, attention_mask=attention_mask)
|
| 189 |
+
x = x + attn_out
|
| 190 |
+
if self.refresh is not None:
|
| 191 |
+
x = self.refresh(x, attn_out, e0, conv_state=conv_state, layer_idx=self.layer_idx)
|
| 192 |
+
x = x + self.mlp(self.ln_2(x))
|
| 193 |
+
return x
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
class GPTS3PreTrainedModel(PreTrainedModel):
|
| 197 |
+
config_class = GPTS3Config
|
| 198 |
+
base_model_prefix = "transformer"
|
| 199 |
+
supports_gradient_checkpointing = False
|
| 200 |
+
|
| 201 |
+
def _init_weights(self, module):
|
| 202 |
+
std = self.config.hidden_size ** -0.5
|
| 203 |
+
if isinstance(module, nn.Linear):
|
| 204 |
+
torch.nn.init.normal_(module.weight, mean=0.0, std=std)
|
| 205 |
+
elif isinstance(module, nn.Embedding):
|
| 206 |
+
torch.nn.init.normal_(module.weight, mean=0.0, std=std)
|
| 207 |
+
elif isinstance(module, nn.Conv1d):
|
| 208 |
+
torch.nn.init.normal_(module.weight, mean=0.0, std=std)
|
| 209 |
+
|
| 210 |
+
|
| 211 |
+
class GPTS3ForCausalLM(GPTS3PreTrainedModel, GenerationMixin):
|
| 212 |
+
_tied_weights_keys = {"lm_head.weight": "transformer.wte.weight"}
|
| 213 |
+
|
| 214 |
+
def __init__(self, config):
|
| 215 |
+
super().__init__(config)
|
| 216 |
+
self.config = config
|
| 217 |
+
self.transformer = nn.ModuleDict(dict(
|
| 218 |
+
wte=nn.Embedding(config.vocab_size, config.hidden_size),
|
| 219 |
+
h=nn.ModuleList([GPTS3Block(config, i) for i in range(config.num_hidden_layers)]),
|
| 220 |
+
ln_f=RMSNorm(config.hidden_size, eps=config.rms_norm_eps),
|
| 221 |
+
))
|
| 222 |
+
self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
|
| 223 |
+
if config.tie_word_embeddings:
|
| 224 |
+
self.lm_head.weight = self.transformer["wte"].weight
|
| 225 |
+
self._rope_cache = None # (cos, sin) cached on device up to some length
|
| 226 |
+
self.post_init()
|
| 227 |
+
|
| 228 |
+
def get_input_embeddings(self):
|
| 229 |
+
return self.transformer["wte"]
|
| 230 |
+
|
| 231 |
+
def set_input_embeddings(self, value):
|
| 232 |
+
self.transformer["wte"] = value
|
| 233 |
+
|
| 234 |
+
def get_output_embeddings(self):
|
| 235 |
+
return self.lm_head
|
| 236 |
+
|
| 237 |
+
def set_output_embeddings(self, new_embeddings):
|
| 238 |
+
self.lm_head = new_embeddings
|
| 239 |
+
|
| 240 |
+
def prepare_inputs_for_generation(self, input_ids, past_key_values=None, attention_mask=None, **kwargs):
|
| 241 |
+
if past_key_values is not None and past_key_values.get_seq_length() > 0:
|
| 242 |
+
input_ids = input_ids[:, -1:]
|
| 243 |
+
return {
|
| 244 |
+
"input_ids": input_ids,
|
| 245 |
+
"attention_mask": attention_mask,
|
| 246 |
+
"past_key_values": past_key_values,
|
| 247 |
+
"use_cache": True,
|
| 248 |
+
}
|
| 249 |
+
|
| 250 |
+
def _get_rope(self, seq_len, device):
|
| 251 |
+
cache = self._rope_cache
|
| 252 |
+
if cache is None or cache[0].device != device or cache[0].size(0) < seq_len:
|
| 253 |
+
cos, sin = precompute_rope_cos_sin(self.config.head_dim, seq_len, self.config.rope_theta)
|
| 254 |
+
cache = (cos.to(device), sin.to(device))
|
| 255 |
+
self._rope_cache = cache
|
| 256 |
+
return cache[0][:seq_len], cache[1][:seq_len]
|
| 257 |
+
|
| 258 |
+
def forward(
|
| 259 |
+
self,
|
| 260 |
+
input_ids,
|
| 261 |
+
attention_mask=None,
|
| 262 |
+
labels=None,
|
| 263 |
+
past_key_values: Optional[DynamicCache] = None,
|
| 264 |
+
use_cache=False,
|
| 265 |
+
**kwargs,
|
| 266 |
+
):
|
| 267 |
+
B, T = input_ids.size()
|
| 268 |
+
conv_state = None
|
| 269 |
+
if use_cache:
|
| 270 |
+
if past_key_values is None:
|
| 271 |
+
past_key_values = DynamicCache()
|
| 272 |
+
# Conv-state cache for the refresh gate lives alongside the KV cache,
|
| 273 |
+
# so it shares the same lifetime as a single generation.
|
| 274 |
+
if not hasattr(past_key_values, "_refresh_conv_state"):
|
| 275 |
+
past_key_values._refresh_conv_state = {}
|
| 276 |
+
conv_state = past_key_values._refresh_conv_state
|
| 277 |
+
|
| 278 |
+
past_len = past_key_values.get_seq_length() if past_key_values is not None else 0
|
| 279 |
+
x = self.transformer["wte"](input_ids)
|
| 280 |
+
e0 = x # original token embeddings, re-injected at every refresh layer
|
| 281 |
+
|
| 282 |
+
cos, sin = self._get_rope(past_len + T, input_ids.device)
|
| 283 |
+
cos, sin = cos[past_len:], sin[past_len:]
|
| 284 |
+
|
| 285 |
+
for block in self.transformer["h"]:
|
| 286 |
+
x = block(
|
| 287 |
+
x, e0, cos, sin,
|
| 288 |
+
past_key_values if use_cache else None,
|
| 289 |
+
use_cache,
|
| 290 |
+
attention_mask=attention_mask,
|
| 291 |
+
conv_state=conv_state,
|
| 292 |
+
)
|
| 293 |
+
|
| 294 |
+
x = self.transformer["ln_f"](x)
|
| 295 |
+
logits = self.lm_head(x)
|
| 296 |
+
|
| 297 |
+
loss = None
|
| 298 |
+
if labels is not None:
|
| 299 |
+
shift_logits = logits[..., :-1, :].contiguous()
|
| 300 |
+
shift_labels = labels[..., 1:].contiguous()
|
| 301 |
+
loss = F.cross_entropy(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
|
| 302 |
+
|
| 303 |
+
return CausalLMOutputWithPast(
|
| 304 |
+
loss=loss,
|
| 305 |
+
logits=logits,
|
| 306 |
+
past_key_values=past_key_values if use_cache else None,
|
| 307 |
+
)
|
| 308 |
+
|
| 309 |
+
|
| 310 |
+
# Backward-compatible aliases for configs/checkpoints created under earlier
|
| 311 |
+
# family names.
|
| 312 |
+
GPTX3ForCausalLM = GPTS3ForCausalLM
|
| 313 |
+
GPTS3ForCausalLM = GPTS3ForCausalLM
|
special_tokens_map.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token": "<bos>",
|
| 3 |
+
"eos_token": "<eos>",
|
| 4 |
+
"unk_token": "<unk>",
|
| 5 |
+
"pad_token": "<eos>"
|
| 6 |
+
}
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": true,
|
| 3 |
+
"added_tokens_decoder": {
|
| 4 |
+
"0": {
|
| 5 |
+
"content": "<unk>",
|
| 6 |
+
"lstrip": false,
|
| 7 |
+
"normalized": false,
|
| 8 |
+
"rstrip": false,
|
| 9 |
+
"single_word": false,
|
| 10 |
+
"special": true
|
| 11 |
+
},
|
| 12 |
+
"1": {
|
| 13 |
+
"content": "<bos>",
|
| 14 |
+
"lstrip": false,
|
| 15 |
+
"normalized": false,
|
| 16 |
+
"rstrip": false,
|
| 17 |
+
"single_word": false,
|
| 18 |
+
"special": true
|
| 19 |
+
},
|
| 20 |
+
"2": {
|
| 21 |
+
"content": "<eos>",
|
| 22 |
+
"lstrip": false,
|
| 23 |
+
"normalized": false,
|
| 24 |
+
"rstrip": false,
|
| 25 |
+
"single_word": false,
|
| 26 |
+
"special": true
|
| 27 |
+
}
|
| 28 |
+
},
|
| 29 |
+
"bos_token": "<bos>",
|
| 30 |
+
"eos_token": "<eos>",
|
| 31 |
+
"unk_token": "<unk>",
|
| 32 |
+
"pad_token": "<eos>",
|
| 33 |
+
"clean_up_tokenization_spaces": false,
|
| 34 |
+
"model_max_length": 512,
|
| 35 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 36 |
+
"vocab_size": 4096
|
| 37 |
+
}
|
usage.py
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 2 |
+
import torch
|
| 3 |
+
|
| 4 |
+
model_name = "AxiomicLabs/GPT-S2-5M"
|
| 5 |
+
|
| 6 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
|
| 7 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 8 |
+
model_name,
|
| 9 |
+
trust_remote_code=True,
|
| 10 |
+
torch_dtype=torch.bfloat16,
|
| 11 |
+
device_map="auto",
|
| 12 |
+
)
|
| 13 |
+
|
| 14 |
+
prompt = "The future of AI is"
|
| 15 |
+
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
| 16 |
+
|
| 17 |
+
with torch.inference_mode():
|
| 18 |
+
output = model.generate(
|
| 19 |
+
**inputs,
|
| 20 |
+
max_new_tokens=120,
|
| 21 |
+
do_sample=True,
|
| 22 |
+
temperature=0.8,
|
| 23 |
+
top_p=0.95,
|
| 24 |
+
repetition_penalty=1.1,
|
| 25 |
+
no_repeat_ngram_size=4,
|
| 26 |
+
)
|
| 27 |
+
|
| 28 |
+
text = tokenizer.decode(output[0], skip_special_tokens=True)
|
| 29 |
+
print(text)
|