Update README.md

README.md

@@ -1,3 +1,75 @@
----
-license: mit
----
+---
+license: mit
+tags:
+- protein
+- antibody
+- nanobody
+- vhh
+- structure
+- masked-language-model
+- esm
+- saprot
+base_model: westlake-repl/SaProt_650M_PDB
+pipeline_tag: fill-mask
+---
+
+# SaNano: Structure-Aware Nanobody Language Model
+
+SaNano is a protein language model fine-tuned on VHH (nanobody) sequences for improved representation learning of single-domain antibodies. Built on top of [SaProt-650M](https://huggingface.co/westlake-repl/SaProt_650M_PDB), this model incorporates both sequence and predicted structure information to generate high-quality embeddings for VHH antibodies.
+
+## Model Description
+
+- **Base Model:** [SaProt_650M_PDB](https://huggingface.co/westlake-repl/SaProt_650M_PDB)
+- **Model Size:** 650M parameters
+- **Hidden Dimensionality:** 1280
+- **Fine-tuning Method:** LoRA (Low-Rank Adaptation)
+  - Rank: 32
+  - Alpha: 64
+  - Dropout: 0.1
+  - Target modules: query, key, value, dense
+- **Training Objective:** Masked Language Modeling (MLM)
+  - Masking probability: 15%
+  - Masking method: Uniform MLM
+
+## Training Data
+
+The model was fine-tuned on a curated dataset of 75% VHH sequences with predicted structures and 25% protein sequences with predicted structures. In training the model sees 50% sequences with structural tokens and 50% sequences with structureal tokens masked.
+
+## Training Configuration
+
+- **Steps:** 5000
+- **Batch Size:** 16 per device (with gradient accumulation steps of 4)
+- **Learning Rate:** 2e-5
+- **Scheduler:** Cosine with 1000 warmup steps
+- **Weight Decay:** 0.01
+- **Precision:** FP16
+
+## Usage
+
+### Basic Usage
+
+```python
+from transformers import EsmForMaskedLM, EsmTokenizer
+
+def add_structure_masking(sequence):
+    return "#".join(sequence) + "#"
+
+model = EsmForMaskedLM.from_pretrained("novonordisk-red/SaNano")
+tokenizer = EsmTokenizer.from_pretrained("novonordisk-red/SaNano")
+
+sequences = [
+    "EVQLVESGGGLVQAGGSLRLSCAASGFTFPTYAMAWFRQAPGKGREFV",
+    "QVQLQESGGGLVQAGGSLRLSCAASGRTFSSYAMGWFRQAPGKEREFVAAI",
+    "EVQLVESGGGLVQPGGSLRLSCAASGFTFSSYAMSWVRQAPGKG",
+    "DVQLVESGGGLVQAGGSLRLSCAASGRTFSTYAMGWFRQAPGKGREFVAGISWS"
+]
+
+# To convert to structure-aware format with masking. This is necessary if structural tokens are not computed
+structure_aware_sequences = [add_structure_masking(seq) for seq in sequences]
+
+# Result example:
+# ["E#V#Q#L#V#E#S#G#G#G#L#V#Q#A#G#G#S#L#R#L#S#C#A#A#S#G#F#T#F#P#T#Y#A#M#A#W#F#R#Q#A#P#G#...", ...]
+
+# For structure-aware input, use '#' between amino acids
+inputs = tokenizer(structure_aware_sequences, return_tensors="pt")
+outputs = model(**inputs)