mistralai

/

Mistral-Medium-3.5-128B

Like 488

Follow

Mistral AI_ 20.3k

Just checking if someone else can confirm this problem.

vllm version 0.20.1rc1.dev64+g5560cac7e

CUDA_VISIBLE_DEVICES=0,1 vllm serve mistralai/Mistral-Medium-3.5-128B --tensor-parallel-size 2   --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral --max_num_batched_tokens 16384 --max_num_seqs 4 --gpu_memory_utilization 0.93 --served-model-name Mistral-Medium-3.5-128B --max-model-len auto --enable-sleep-mode --speculative_config '{
    "model": "mistralai/Mistral-Medium-3.5-128B-EAGLE",
    "num_speculative_tokens": 3,
    "method": "eagle",
    "max_model_len": "65536"
  }'

And similar configs, I end up with at most like 100k kv-cache allocation (.97 utilization) but it's only using 84gb per card.

Anyway, thanks Mistral - model looks great!