16000 vocab size tokenizer padded to 16384, trained on the following data:

DATASETS = [
    ("mlfoundations/dclm-baseline-1.0-parquet", None, "train", "text", 75000),
    ("HuggingFaceFW/finepdfs-edu", "eng_Latn", "train", "text", 5000),
    ("openbmb/Ultra-FineWeb", None, "zh", "content", 25000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "cpp", "content", 10000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "js", "content", 10000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "py", "content", 10000),
    ("openbmb/UltraData-Code", "UltraData-Code-L2", "rust", "content", 10000),
    ("openbmb/UltraData-Math", "UltraData-Math-L2-preview", "train", "content", 20000),
]

Code is in notebook.py

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

allura-forge
/

Rambley-Tokenizer-v1-16k

Datasets used to train allura-forge/Rambley-Tokenizer-v1-16k