Multi-Vector Encoder

This is a Multi-Vector Encoder model trained on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

Model Details

Model Description

  • Model Type: Multi-Vector Encoder
  • Maximum Sequence Length: 8192 tokens
  • Output Dimensionality: 128 dimensions
  • Similarity Function: MaxSim
  • Supported Modality: Text
  • Training Dataset:
    • msmarco-bm25

Model Sources

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_expansion': {'strategy': 'min', 'attend': False, 'token': None, 'length': 32}, 'architecture': 'ModernBertModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': [], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/mve-ModernBERT-base-token-pooling-baseline")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'when was the first computer launched',
]
documents = [
    'The IBM 610 was the first personal computer, and it was launched in 1957. 3 people found this useful.',
    'The IBM 610 was the first personal computer, and it was launched in 1957. 3 people found this useful.',
    'Tron Price Chart Bitcoin (TRX/BTC) Tron price for today is ฿0.00000417. It has a current circulating supply of 65.7 Billion coins and a total volume exchanged of ฿15,283',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (32, 128) (24, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[23.3548, 23.3548,  7.8494]])

Training Details

Training Dataset

msmarco-bm25

  • Dataset: msmarco-bm25
  • Size: 120,000 training samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 100 samples:
    query positive negative
    type string string string
    modality text text text
    details
    • min: 5 tokens
    • mean: 9.2 tokens
    • max: 17 tokens
    • min: 31 tokens
    • mean: 82.7 tokens
    • max: 187 tokens
    • min: 19 tokens
    • mean: 83.45 tokens
    • max: 184 tokens
  • Samples:
    query positive negative
    cost to install hardwood floor per room How Much Does Hardwood Floors Cost? Installing a standard solid-strip hardwood floor averages about $8 a square foot for materials, insulation and site finishing, and up to $12 a square foot for wide pine planks, or $1,150-$1,750 for a 12x12-foot room. Factory-finished wood starts at $8 a square foot installed, but generally runs $10-$14, or $1,140-$2,000 for a 12x12 space. Cost to Install a Hardwood Floor. 1 Share. 2 Hiring Local Professionals. 3 Sponsored Services. 4 Cost to Install a Hardwood Floor - Notes and General Information. 5 Cost to Install a Hardwood Floor - Unit Price References. 6 Hardwood Floor Installation - Average Cost Per Square Foot. 7 Help the homewyse Community.
    football colleges in georgia Four-Year Georgia Universities or Colleges With Football Teams. There are 735 four-year colleges and universities that field football teams in the United States as of 2011. Six hundred thirty-nine belong to the National Collegiate Athletic Association's various divisions and 96 to the National Association of Intercollegiate Athletics. NCAA Division II (with football) Colleges in Georgia. If you're looking for Georgia NCAA Division II (with football) literacy rates, CALiteracy.org can help you find the schools with the most to offer. The information found here provides state literacy rates and information for NCAA Division II (with football) colleges across Georgia.
    what does glucosamine sulphate do Glucosamine sulfate is a chemical found in the human body. It is used by the body to produce a variety of other chemicals that are involved in building tendons, ligaments, cartilage, and the thick fluid that surrounds joints. Joints are cushioned by the fluid and cartilage that surround them.or example, the glucosamine sulfate that is put into dietary supplements is often harvested from the shells of shellfish. Glucosamine sulfate used in dietary supplements does not always come from natural sources. It can also be made in a laboratory. “Glucosamine sulphate, which is stabilized with sodium chloride or table salt, can contain as much as 30% sodium,” Keith says. “This is a massive consideration for people who want to reduce their dietary intake of sodium (salt).” Glucosamine HCL and glucosamine sulphate 2KCL do not contain sodium chloride.
  • Loss: CachedMultiVectorMultipleNegativesRankingLoss with these parameters:
    {
        "scale": 1.0,
        "similarity_fct": "colbert_scores",
        "mini_batch_size": 64,
        "mini_batch_num_tokens": null,
        "score_mini_batch_size": 64,
        "gather_across_devices": false
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 256
  • num_train_epochs: 1
  • learning_rate: 3e-05
  • warmup_steps: 0.05
  • bf16: True
  • data_seed: 42
  • dataloader_pin_memory: False
  • batch_sampler: no_duplicates
  • max_length: 300

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 256
  • num_train_epochs: 1
  • max_steps: -1
  • learning_rate: 3e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.05
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: trackio
  • per_device_eval_batch_size: 8
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: 42
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_pin_memory: False
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: []
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • warmup_ratio: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • max_length: 300

Training Logs

Epoch Step Training Loss
0.0213 10 5.6879
0.0426 20 4.1837
0.0640 30 1.8269
0.0853 40 1.2324
0.1066 50 0.9383
0.1279 60 0.7743
0.1493 70 0.7114
0.1706 80 0.6752
0.1919 90 0.6292
0.2132 100 0.5999
0.2345 110 0.5819
0.2559 120 0.5494
0.2772 130 0.5547
0.2985 140 0.5337
0.3198 150 0.5358
0.3412 160 0.4992
0.3625 170 0.5282
0.3838 180 0.4945
0.4051 190 0.4888
0.4264 200 0.5008
0.4478 210 0.4672
0.4691 220 0.4914
0.4904 230 0.4869
0.5117 240 0.4635
0.5330 250 0.4549
0.5544 260 0.4384
0.5757 270 0.4399
0.5970 280 0.4514
0.6183 290 0.4665
0.6397 300 0.4336
0.6610 310 0.3887
0.6823 320 0.4455
0.7036 330 0.4244
0.7249 340 0.4502
0.7463 350 0.4426
0.7676 360 0.4337
0.7889 370 0.4406
0.8102 380 0.4238
0.8316 390 0.4164
0.8529 400 0.4628
0.8742 410 0.4301
0.8955 420 0.4459
0.9168 430 0.4190
0.9382 440 0.4197
0.9595 450 0.4086
0.9808 460 0.4402

Training Time

  • Training: 27.6 minutes

Framework Versions

  • Python: 3.12.3
  • Sentence Transformers: 6.3.0.dev0
  • Transformers: 5.4.0
  • PyTorch: 2.11.0+cu128
  • Accelerate: 1.13.0
  • Datasets: 4.8.5
  • Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiVectorMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
Downloads last month
51
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Papers for tomaarsen/mve-ModernBERT-base-token-pooling-baseline

Article mentioning tomaarsen/mve-ModernBERT-base-token-pooling-baseline