--- base_model: gpt2 datasets: - wikimedia/wikipedia library_name: Distily license: mit tags: - bitnet - 1.58b - generated_from_trainer model-index: - name: distily_multi_attn_experiment_ortho results: [] --- # Summary Distilled with [Distily](https://github.com/lapp0/distily) library using teacher model [gpt2](https://huggingface.co/gpt2) on dataset [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia). # Model Architecture: - **Architecture**: `GPT2LMHeadModel` - **Total Parameters**: 124,439,808 - **Data Type (dtype)**: torch.bfloat16 - **Model Size**: 0.24 GB # Benchmark Metrics Comparison | Metric | attn_layer_mapper=all, attn_loss_fn=cos, attn_projector=orthogonal, attn_weight=5 | attn_layer_mapper=layer-2, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=25.0 | teacher | | :--- | :--- | :--- | :--- | | ai2_arc (acc) | 0.313 | 0.305 | 0.354 | | ai2_arc (acc_norm) | 0.31 | 0.302 | 0.339 | | arc_challenge (acc) | 0.181 | 0.173 | 0.188 | | arc_challenge (acc_norm) | 0.224 | 0.223 | 0.222 | | arc_easy (acc) | 0.378 | 0.37 | 0.436 | | arc_easy (acc_norm) | 0.353 | 0.34 | 0.396 | | boolq (acc) | 0.49 | 0.387 | 0.51 | | cola (mcc) | -0.041 | 0.044 | 0.01 | | glue (acc) | 0.396 | 0.412 | 0.403 | | glue (f1) | 0.516 | 0.451 | 0.529 | | glue (mcc) | -0.041 | 0.044 | 0.01 | | hellaswag (acc) | 0.32 | 0.315 | 0.343 | | hellaswag (acc_norm) | 0.348 | 0.344 | 0.393 | | mnli (acc) | 0.336 | 0.338 | 0.338 | | mnli_mismatch (acc) | 0.343 | 0.351 | 0.346 | | mrpc (acc) | 0.444 | 0.353 | 0.515 | | mrpc (f1) | 0.478 | 0.143 | 0.631 | | qnli (acc) | 0.488 | 0.497 | 0.491 | | qqp (acc) | 0.356 | 0.406 | 0.367 | | qqp (f1) | 0.522 | 0.501 | 0.512 | | rte (acc) | 0.56 | 0.549 | 0.516 | | sst2 (acc) | 0.498 | 0.545 | 0.511 | | wikitext (bits_per_byte) | 1.118 | 1.127 | 0.98 | | wikitext (byte_perplexity) | 2.17 | 2.184 | 1.973 | | wikitext (word_perplexity) | 63.05 | 65.25 | 37.82 | | wnli (acc) | 0.408 | 0.451 | 0.451 | # Resource Usage Comparison - VRAM Use: 8.2855 GB # Distillation (Teacher -> Student) Architecture Difference: - **Architecture**: `GPT2LMHeadModel` -> `GPT2LMHeadModel` - **Total Parameters**: 124,439,808 -> 124,439,808 - **Data Type (dtype)**: torch.bfloat16 -> torch.bfloat16 - **Model Size**: 0.24 GB -> 0.24 GB
Module Diff Details ```diff ```

# Train Dataset Trained on 145,724,804 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset. - Num Samples: `247,500` - Subset: `20231101.en` - Split: `train` # Training Objective ``` DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=5, loss_fn=cos, layer_mapper=all)) ``` # Hyperparameters The following hyperparameters were used during training:
Expand - learning_rate: `0.0001` - train_batch_size: `4` - eval_batch_size: `8` - seed: `42` - optimizer: `Adam with betas=(0.9,0.999) and epsilon=1e-08` - lr_scheduler_type: `cosine_with_min_lr` - lr_scheduler_warmup_ratio: `0.5` - num_epochs: `1.0` - distillation_objective: `DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl), attn_loss_component=LossComponent(label=attn, weight=5, loss_fn=cos, layer_mapper=all))` - train_embeddings: `True` - lr_scheduler: `` - student_model_name_or_path: `None` - student_config_name_or_path: `None` - student_model_config: `None` - reinitialize_weights: `None` - copy_teacher_modules: `[('lm_head', False)]` - student_model_as_bitnet: `True` - student_model_compile: `False` - dropout: `None` - teacher_model_name_or_path: `gpt2` - teacher_load_in_8bit: `False` - teacher_load_in_4bit: `False` - teacher_model_compile: `False` - dataset_uri: `wikimedia/wikipedia` - dataset_subset: `20231101.en` - dataset_split: `train` - dataset_column_name: `text` - dataset_sample_size: `250000` - dataset_test_size: `0.01` - gradient_accumulation_steps: `1` - weight_decay: `0.0` - max_grad_norm: `1.0` - warmup_ratio: `0.5` - warmup_steps: `0` - gradient_checkpointing: `True`

# Framework Versions - Distily 0.3.0 - Transformers 4.44.0 - Pytorch 2.3.0 - Datasets 2.21.0