voight-kampff-pan2024-gte-en-v1.5

A fine-tuned sentence embedding model for AI-generated text detection, built for the Voight-Kampff Generative AI Shared Task (PAN @ CLEF 2024).

This model generates embeddings where human-written texts cluster together and are separated from AI-generated texts in the vector space. It is designed to be paired with a downstream classifier (we use a calibrated Linear SVM) for authorship verification.

Model Details

Parameter	Value
Base Model	Alibaba-NLP/gte-base-en-v1.5
Fine-tuning Method	Contrastive learning with triplet loss
Frozen Layers	10 of 12 (only last 2 fine-tuned)
Epochs	20
Batch Size	16
Learning Rate	1e-5
Weight Decay	0.01
Gradient Accumulation	2 steps
Warmup	10% of total training steps
Training Platform	Kaggle

Training Data

Trained on the PAN 2024 competition dataset, augmented with texts rewritten by local LLMs via Ollama (llama 3.2 1b, qwen 2.5 1b, gemma 2 2b).

• Texts are chunked into ~64 token segments using the gte-base-en-v1.5 tokenizer
• Triplets formed as (human anchor, human positive, AI negative)
• 15% of training samples include leetspeak noise injection via pyleetspeak
• AI data from 18 different LLM sources (GPT-3.5, GPT-4, LLaMA, Mistral, Alpaca, Qwen, Gemma, etc.)

Usage

Generating Embeddings

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    'alejandroparbas/voight-kampff-pan2024-gte-en-v1.5',
    trust_remote_code=True
)

texts = [
    "A chunk of human-written text...",
    "A chunk of AI-generated text..."
]

embeddings = model.encode(texts)

Training a Classifier on the Embeddings

The embeddings can be used to train any classifier. We use a calibrated Linear SVM:

from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV

# Generate embeddings for your labeled data
train_embeddings = model.encode(train_texts, show_progress_bar=True)
test_embeddings = model.encode(test_texts, show_progress_bar=True)

# Train SVM
svm = LinearSVC(random_state=42, max_iter=10000)
svm.fit(train_embeddings, train_labels)  # labels: 0 = human, 1 = AI

# Calibrate for probability output
calibrated_svm = CalibratedClassifierCV(estimator=svm, cv='prefit')
calibrated_svm.fit(test_embeddings, test_labels)

Full Classification Pipeline

For the complete text-pair classification pipeline (chunking, embedding, SVM, scoring), see the GitHub repository.

A pre-trained SVM classifier is also available: alejandroparbas/voight-kampff-pan2024-classifier

Evaluation Results

When paired with a calibrated Linear SVM, the full pipeline achieves:

Chunk-Level Classification (~64 token chunks)

Metric	Score
F1	~0.80
Accuracy	~0.80

Full Text-Pair Classification (with chunk averaging)

On PAN 2024 test split (with noise):

Metric	Score
ROC-AUC	0.993
Brier	0.924
C@1	0.951
F1	0.951
F0.5u	0.953
Mean	0.955

On external Kaggle AI vs Human Text dataset:

Metric	Score
ROC-AUC	0.948
Mean	0.891

Comparison with PAN 2024 Competition Leaderboard

#	Team	ROC-AUC	Brier	C@1	F1	F0.5u	Mean
1	marsan	0.961	0.928	0.912	0.884	0.932	0.924
2	you-shun-you-de	0.931	0.926	0.928	0.905	0.913	0.921
3	baselineavengers	0.925	0.869	0.882	0.875	0.869	0.886
-	Baseline	0.751	0.780	0.734	0.720	0.720	0.741

Note: Our results are evaluated on our own test split and are not directly comparable to the official competition leaderboard.

Training Metrics

Epoch	Step	Training Loss	Validation Loss	Cosine Accuracy
2.7778	500	5.0051	5.0043	0.4734
5.5556	1000	5.0008	4.9969	0.5029
8.3333	1500	4.9902	4.9804	0.5650
11.1111	2000	4.953	4.8645	0.7361
13.8889	2500	4.7692	4.5741	0.8066
16.6667	3000	4.4897	4.2532	0.8212
19.4444	3500	4.2394	4.0644	0.8324

Limitations

• English only: The base model and training data are in English. Performance on other languages is not guaranteed.
• Short texts: Works best on texts long enough to produce multiple ~64 token chunks. With only 1-2 chunks, downstream classification accuracy is ~80%.
• Requires downstream classifier: This model produces embeddings, not classifications directly.
• Evaluation caveat: Results are on our own test split, not the official PAN 2024 evaluation set.

Authors

• Alejandro Pardo Bascuñana - Universidad Politécnica de Madrid
• Pedro Amaya Moreno - Universidad Politécnica de Madrid

Developed as part of the NLP course in the Master's program Aprendizaje Automático y Datos Masivos at UPM (2024-2025).

Citation

@misc{pardo2025voightkampff,
  title={Voight-Kampff: Contrastive Embedding Learning for AI-Generated Text Detection},
  author={Pardo-Bascu{\~n}ana, Alejandro and Amaya-Moreno, Pedro},
  year={2025},
  url={https://github.com/Alejandro-Pardo/voight-kampff-pan2024/}
}

Links

• GitHub: https://github.com/Alejandro-Pardo/voight-kampff-pan2024/
• Classifier Model: alejandroparbas/voight-kampff-pan2024-classifier
• Competition: PAN @ CLEF 2024 - Generative AI Authorship Verification
• Base Model: Alibaba-NLP/gte-base-en-v1.5