ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
🔄 In a Training Loop
Daniel Gallagher
DanielGallagherIRE
AI & ML interests
NLP Researcher at InfAI in Leipzig, Germany.
Recent Activity
updated a dataset about 3 hours ago
DanielGallagherIRE/fineweb-edu-ga-20261015-193204 updated a dataset about 3 hours ago
DanielGallagherIRE/fineweb-edu-ga-20261015-193206 updated a dataset about 3 hours ago
DanielGallagherIRE/fineweb-edu-ga-20261015-192857Organizations
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 1.03k -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 1.56k -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 1.56k -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 2.14k
ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 1.03k -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 1.56k -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 1.56k -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 2.14k
models 54
DanielGallagherIRE/ModernBERT-O3-Shuffling-SENT
0.2B • Updated • 67
DanielGallagherIRE/ModernBERT-O3-Shuffling-DOC
0.2B • Updated • 73
DanielGallagherIRE/ModernBERT-O3-Baseline
0.2B • Updated • 82
DanielGallagherIRE/ModernBERT-O3-PMI-5
0.2B • Updated • 47
DanielGallagherIRE/ModernBERT-O3-PMI-2.5
0.2B • Updated • 47
DanielGallagherIRE/ModernBERT-O3-PMI-10
0.2B • Updated • 60
DanielGallagherIRE/ModernBERT-O3-Nouns-POS
0.2B • Updated • 54
DanielGallagherIRE/ModernBERT-O3-PMI-7.5
0.2B • Updated • 65
DanielGallagherIRE/ModernBERT-O3-PMI-12.5
0.2B • Updated • 61
DanielGallagherIRE/ModernBERT-O3-Nouns-DEL
0.2B • Updated • 71
datasets 35
DanielGallagherIRE/fineweb-edu-ga-20261015-193204
Viewer • Updated • 80.1k
DanielGallagherIRE/fineweb-edu-ga-20261015-193206
Viewer • Updated • 80.1k
DanielGallagherIRE/fineweb-edu-ga-20261015-192857
Viewer • Updated • 80.1k
DanielGallagherIRE/fineweb-edu-ga-20261015-192214
Viewer • Updated • 80.1k
DanielGallagherIRE/fineweb-edu-ga-20261015-192405
Viewer • Updated • 80.1k
DanielGallagherIRE/fineweb-edu-ga-20261015-190617
Viewer • Updated • 400k
DanielGallagherIRE/fineweb-edu-ga-20261015-094928
Viewer • Updated • 100k
DanielGallagherIRE/fineweb-edu-ga-20261015-093119
Updated
DanielGallagherIRE/FineWeb-Edu-100M-Irish
Viewer • Updated • 100k • 279
DanielGallagherIRE/fineweb-edu-ga-20261007-155449
Viewer • Updated • 180 • 25