---
license: mit
---
# Shakespeare Text Generator 🎭

A character-level transformer language model trained from scratch on Shakespeare's plays.

## Model Description

This is a **learning project** demonstrating how to train a transformer-based language model from scratch using PyTorch. The model was trained on Shakespeare's complete works and can generate text in a similar style.

### Key Details

- **Model Type:** Character-level Transformer Language Model
- **Architecture:** 6-layer Transformer Encoder with causal masking
- **Parameters:** ~4M parameters
- **Training Data:** Shakespeare's plays (~1.1M characters)
- **Framework:** PyTorch
- **Training Time:** ~8 hours on single GPU
- **Purpose:** Educational/Learning Project

## Model Architecture

```
- Vocabulary Size: 65 characters
- Embedding Dimension: 256
- Number of Attention Heads: 8
- Number of Layers: 6
- Sequence Length: 128 characters
- Feedforward Dimension: 1024
- Dropout: 0.2
```

## Training Details

### Training Data

- **Source:** Shakespeare's complete works from [Karpathy's char-rnn repository](https://github.com/karpathy/char-rnn)
- **Size:** ~1.1M characters
- **Preprocessing:** Minimal - character-level tokenization only

### Training Procedure

- **Epochs:** 25
- **Batch Size:** 64
- **Learning Rate:** 0.001 (Adam optimizer)
- **Scheduler:** Cosine Annealing
- **Hardware:** NVIDIA GPU
- **Final Training Loss:** 0.937
- **Final Validation Loss:** 0.629
- **Validation Accuracy:** 79.9%
- **Perplexity:** 1.9

### Training Configuration

```python
optimizer = Adam(lr=0.001, weight_decay=1e-4)
scheduler = CosineAnnealingLR(T_max=25)
criterion = CrossEntropyLoss()
```

## Usage

### Installation

```bash
pip install torch numpy
```

### Basic Usage

```python
import torch
import torch.nn as nn

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Load model
checkpoint = torch.load('best_model.pth', map_location=device)
tokenizer = checkpoint['tokenizer']
model = checkpoint['model']  # or recreate from architecture

# Generate text
def generate(prompt, max_length=300, temperature=0.8):
    model.eval()
    indices = tokenizer.encode(prompt)
    
    with torch.no_grad():
        for _ in range(max_length):
            x = torch.tensor(indices[-128:]).unsqueeze(0).to(device)
            if x.shape[1] < 128:
                padding = torch.zeros(1, 128 - x.shape[1]).to(device)
                x = torch.cat([padding, x], dim=1)
            
            logits = model(x)
            logits = logits[0, -1, :] / temperature
            probs = torch.softmax(logits, dim=-1)
            next_idx = torch.multinomial(probs, num_samples=1).item()
            indices.append(next_idx)
    
    return tokenizer.decode(indices)

# Example
text = generate("ROMEO:\n", max_length=200)
print(text)
```

### Example Outputs

**Prompt:** `"ROMEO:"`

**Output:**
```
ROMEO:
What light through yonder window breaks?
'Tis the sun, and Juliet is the east...
```

**Note:** The model generates Shakespeare-style text but may include some inconsistencies or unusual patterns due to its small size and character-level tokenization.

## Limitations

⚠️ **This is a learning/educational project. The model has significant limitations:**

### Quality Limitations
- **Small model size** (~1M parameters vs. GPT-2's 117M)
- **Character-level tokenization** (inefficient compared to subword tokenization)
- **Limited context** (128 characters only)
- **Training dataset size** (only Shakespeare's works)
- **No pre-training** (trained from random initialization)

### Expected Behavior
- ✅ Generates text with Shakespeare-like structure
- ✅ Understands character dialogue format
- ✅ Captures some writing style patterns
- ❌ May produce gibberish characters occasionally
- ❌ Limited coherence over long sequences
- ❌ Not suitable for production use

### What This Model Is NOT
- ❌ Not comparable to GPT-2, GPT-3, or modern LLMs (GPT-2 Small has 117M, ~30x larger)
- ❌ Not fine-tuned for instruction following
- ❌ Not suitable for serious text generation applications
- ❌ Not production-ready

## What I Learned

This project was an educational exercise in:

✅ **Transformer Architecture**
- Implementing multi-head attention from scratch
- Understanding positional encodings
- Working with causal masking

✅ **Training Deep Learning Models**
- Setting up training loops in PyTorch
- Using learning rate schedulers
- Monitoring training/validation metrics
- GPU acceleration with CUDA

✅ **Language Modeling**
- Character-level tokenization
- Next-token prediction
- Autoregressive text generation
- Temperature sampling

✅ **MLOps Basics**
- Model checkpointing
- Experiment tracking
- Model evaluation
- Deploying to Hugging Face

## Performance Metrics

| Metric | Value |
|--------|-------|
| Training Loss | 0.937 |
| Validation Loss | 0.629 |
| Validation Accuracy | 79.9% |
| Perplexity | 1.9 |
| Character Error Rate | ~20% |

## Comparison to Baselines

| Model | Parameters | Quality |
|-------|------------|---------|
| This Model | 4M | Low (educational) |
| GPT-2 Small | 117M | High |
| GPT-3 | 175B | Very High |

## Future Improvements

If I were to improve this model, I would:

1. **Use word-level or BPE tokenization** instead of character-level
2. **Increase model size** to 10M+ parameters
3. **Train on more diverse data** (books, articles, etc.)
4. **Pre-train then fine-tune** instead of training from scratch
5. **Use longer context windows** (512+ tokens)
6. **Implement better sampling strategies** (beam search, nucleus sampling)

## Files Included

- `best_model.pth` - Trained model weights
- `training_data.txt` - Shakespeare corpus used for training
- `train_text_model.py` - Training script
- `generate_text.py` - Generation script
- `README.md` - This file

## Reproducibility

To reproduce this model:

```bash
# Clone the repository
git clone [your-repo-url]
cd shakespeare-transformer

# Install dependencies
pip install torch numpy tqdm matplotlib

# Run training
python train_text_model.py

# Generate text
python generate_text.py
```

## Citation

If you use this project for learning purposes, please cite:

```bibtex
@misc{shakespeare-transformer-2024,
  author = {Your Name},
  title = {Shakespeare Text Generator: A Learning Project},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/your-username/shakespeare-transformer}}
}
```

## Acknowledgments

- **Data Source:** [Karpathy's char-rnn repository](https://github.com/karpathy/char-rnn)
- **Inspiration:** Andrej Karpathy's blog posts on RNNs and language modeling
- **Framework:** PyTorch team for the excellent deep learning framework

## License

This project is for educational purposes. The Shakespeare text is in the public domain.


---

**Disclaimer:** This is a learning project created to understand transformer architectures and language modeling. It is not intended for production use or to compete with modern LLMs. The model has significant limitations and is shared purely for educational purposes.

## Tags

`shakespeare` `text-generation` `transformer` `pytorch` `character-level` `educational` `language-model` `from-scratch` `learning-project`