Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
Paper • 2101.06983 • Published • 3
How to use tomaarsen/mve-ModernBERT-base-token-pooling-baseline with sentence-transformers:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("tomaarsen/mve-ModernBERT-base-token-pooling-baseline")
queries = ["Which planet is known as the Red Planet?"]
documents = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)This is a Multi-Vector Encoder model trained on the msmarco-bm25 dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.
MultiVectorEncoder(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_expansion': {'strategy': 'min', 'attend': False, 'token': None, 'length': 32}, 'architecture': 'ModernBertModel'})
(1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
(2): MultiVectorMask({'skiplist_words': [], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
(3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import MultiVectorEncoder
# Download from the 🤗 Hub
model = MultiVectorEncoder("tomaarsen/mve-ModernBERT-base-token-pooling-baseline")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
'when was the first computer launched',
]
documents = [
'The IBM 610 was the first personal computer, and it was launched in 1957. 3 people found this useful.',
'The IBM 610 was the first personal computer, and it was launched in 1957. 3 people found this useful.',
'Tron Price Chart Bitcoin (TRX/BTC) Tron price for today is ฿0.00000417. It has a current circulating supply of 65.7 Billion coins and a total volume exchanged of ฿15,283',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (32, 128) (24, 128)
# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[23.3548, 23.3548, 7.8494]])
query, positive, and negative| query | positive | negative | |
|---|---|---|---|
| type | string | string | string |
| modality | text | text | text |
| details |
|
|
|
| query | positive | negative |
|---|---|---|
cost to install hardwood floor per room |
How Much Does Hardwood Floors Cost? Installing a standard solid-strip hardwood floor averages about $8 a square foot for materials, insulation and site finishing, and up to $12 a square foot for wide pine planks, or $1,150-$1,750 for a 12x12-foot room. Factory-finished wood starts at $8 a square foot installed, but generally runs $10-$14, or $1,140-$2,000 for a 12x12 space. |
Cost to Install a Hardwood Floor. 1 Share. 2 Hiring Local Professionals. 3 Sponsored Services. 4 Cost to Install a Hardwood Floor - Notes and General Information. 5 Cost to Install a Hardwood Floor - Unit Price References. 6 Hardwood Floor Installation - Average Cost Per Square Foot. 7 Help the homewyse Community. |
football colleges in georgia |
Four-Year Georgia Universities or Colleges With Football Teams. There are 735 four-year colleges and universities that field football teams in the United States as of 2011. Six hundred thirty-nine belong to the National Collegiate Athletic Association's various divisions and 96 to the National Association of Intercollegiate Athletics. |
NCAA Division II (with football) Colleges in Georgia. If you're looking for Georgia NCAA Division II (with football) literacy rates, CALiteracy.org can help you find the schools with the most to offer. The information found here provides state literacy rates and information for NCAA Division II (with football) colleges across Georgia. |
what does glucosamine sulphate do |
Glucosamine sulfate is a chemical found in the human body. It is used by the body to produce a variety of other chemicals that are involved in building tendons, ligaments, cartilage, and the thick fluid that surrounds joints. Joints are cushioned by the fluid and cartilage that surround them.or example, the glucosamine sulfate that is put into dietary supplements is often harvested from the shells of shellfish. Glucosamine sulfate used in dietary supplements does not always come from natural sources. It can also be made in a laboratory. |
âGlucosamine sulphate, which is stabilized with sodium chloride or table salt, can contain as much as 30% sodium,â Keith says. âThis is a massive consideration for people who want to reduce their dietary intake of sodium (salt).â Glucosamine HCL and glucosamine sulphate 2KCL do not contain sodium chloride. |
CachedMultiVectorMultipleNegativesRankingLoss with these parameters:{
"scale": 1.0,
"similarity_fct": "colbert_scores",
"mini_batch_size": 64,
"mini_batch_num_tokens": null,
"score_mini_batch_size": 64,
"gather_across_devices": false
}
per_device_train_batch_size: 256num_train_epochs: 1learning_rate: 3e-05warmup_steps: 0.05bf16: Truedata_seed: 42dataloader_pin_memory: Falsebatch_sampler: no_duplicatesmax_length: 300per_device_train_batch_size: 256num_train_epochs: 1max_steps: -1learning_rate: 3e-05lr_scheduler_type: linearlr_scheduler_kwargs: Nonewarmup_steps: 0.05optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 1average_tokens_across_devices: Truemax_grad_norm: 1.0label_smoothing_factor: 0.0bf16: Truefp16: Falsebf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Falseproject: huggingfacetrackio_space_id: trackioper_device_eval_batch_size: 8prediction_loss_only: Trueeval_on_start: Falseeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Falsehub_private_repo: Nonehub_model_id: Nonehub_strategy: every_savehub_always_push: Falsehub_revision: Noneload_best_model_at_end: Falseignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: 42use_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Falsedataloader_num_workers: 0dataloader_pin_memory: Falsedataloader_persistent_workers: Falsedataloader_prefetch_factor: Noneremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_backend: Noneddp_timeout: 1800fsdp: []fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}deepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonewarmup_ratio: Nonelocal_rank: -1prompts: Nonebatch_sampler: no_duplicatesmulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}max_length: 300| Epoch | Step | Training Loss |
|---|---|---|
| 0.0213 | 10 | 5.6879 |
| 0.0426 | 20 | 4.1837 |
| 0.0640 | 30 | 1.8269 |
| 0.0853 | 40 | 1.2324 |
| 0.1066 | 50 | 0.9383 |
| 0.1279 | 60 | 0.7743 |
| 0.1493 | 70 | 0.7114 |
| 0.1706 | 80 | 0.6752 |
| 0.1919 | 90 | 0.6292 |
| 0.2132 | 100 | 0.5999 |
| 0.2345 | 110 | 0.5819 |
| 0.2559 | 120 | 0.5494 |
| 0.2772 | 130 | 0.5547 |
| 0.2985 | 140 | 0.5337 |
| 0.3198 | 150 | 0.5358 |
| 0.3412 | 160 | 0.4992 |
| 0.3625 | 170 | 0.5282 |
| 0.3838 | 180 | 0.4945 |
| 0.4051 | 190 | 0.4888 |
| 0.4264 | 200 | 0.5008 |
| 0.4478 | 210 | 0.4672 |
| 0.4691 | 220 | 0.4914 |
| 0.4904 | 230 | 0.4869 |
| 0.5117 | 240 | 0.4635 |
| 0.5330 | 250 | 0.4549 |
| 0.5544 | 260 | 0.4384 |
| 0.5757 | 270 | 0.4399 |
| 0.5970 | 280 | 0.4514 |
| 0.6183 | 290 | 0.4665 |
| 0.6397 | 300 | 0.4336 |
| 0.6610 | 310 | 0.3887 |
| 0.6823 | 320 | 0.4455 |
| 0.7036 | 330 | 0.4244 |
| 0.7249 | 340 | 0.4502 |
| 0.7463 | 350 | 0.4426 |
| 0.7676 | 360 | 0.4337 |
| 0.7889 | 370 | 0.4406 |
| 0.8102 | 380 | 0.4238 |
| 0.8316 | 390 | 0.4164 |
| 0.8529 | 400 | 0.4628 |
| 0.8742 | 410 | 0.4301 |
| 0.8955 | 420 | 0.4459 |
| 0.9168 | 430 | 0.4190 |
| 0.9382 | 440 | 0.4197 |
| 0.9595 | 450 | 0.4086 |
| 0.9808 | 460 | 0.4402 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{gao2021scaling,
title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
year={2021},
eprint={2101.06983},
archivePrefix={arXiv},
primaryClass={cs.LG}
}