| docker run --rm --name sg-http --publish 127.0.0.1:8710:8000 --device /dev/tenstorrent:/dev/tenstorrent -v /dev/hugepages:/dev/hugepages -v /dev/hugepages-1G:/dev/hugepages-1G -v /var/tmp/gemma4-12b/native/p3d-pc-gate-s1/gemma-4-12B-it:/model:ro -v /var/tmp/gemma4-12b/serve-cache:/cache --workdir /home/container_app_user/tt-metal --entrypoint /home/container_app_user/tt-metal/python_env/bin/python -v /var/tmp/gemma4-12b/native/p3d-pc-gate-s1/gemma-4-12B-it-assistant:/drafter:ro -e GEMMA4_DECODE_KV_ROWS=1 -e GEMMA4_DECODE_MM_DN=1 -e GEMMA4_FUSE_GELU_MUL=1 -e GEMMA4_NATIVE=/model -e GEMMA4_SINGLE_USER=1 -e GEMMA4_VERIFY_SDPA=batched -e GEMMA_PREFIX_SNAPSHOTS=4 -e HF_HUB_OFFLINE=1 -e HF_MODEL=/model -e TORCHDYNAMO_DISABLE=1 -e TRANSFORMERS_OFFLINE=1 -e TT_METAL_CACHE=/cache/6677ccdf80ee229b -e VLLM_TARGET_DEVICE=tt gemma4-12b:p3d-pc-gate-s1 -m vllm.entrypoints.openai.api_server --model /model --served-model-name google/gemma-4-12B-it --host 0.0.0.0 --port 8000 --max-model-len 262144 --max-num-seqs 1 --block-size 64 --max-num-batched-tokens 16384 --enable-chunked-prefill --enable-prefix-caching --seed 0 --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 --additional-config '{"tt": {"trace_mode": "all", "enable_model_warmup": true, "trace_region_size": 200000000}}' --speculative-config '{"method": "gemma4_assistant", "model": "/drafter", "num_speculative_tokens": 5}' |
| INFO 10-05 23:03:35 [__init__.py:44] Available plugins for group vllm.platform_plugins: |
| INFO 10-05 23:03:35 [__init__.py:46] - tt -> vllm_tt_plugin.entrypoints:platform_plugin |
| INFO 10-05 23:03:35 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load. |
| INFO 10-05 23:03:36 [__init__.py:212] Platform plugin tt is activated |
| INFO 10-05 23:03:39 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. |
| INFO 10-05 23:03:39 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. |
| (APIServer pid=1) INFO 10-05 23:03:40 [utils.py:287]
|
| (APIServer pid=1) INFO 10-05 23:03:40 [utils.py:287] █ █ █▄ ▄█
|
| (APIServer pid=1) INFO 10-05 23:03:40 [utils.py:287] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.1.dev14187+g03fa3af2e
|
| (APIServer pid=1) INFO 10-05 23:03:40 [utils.py:287] █▄█▀ █ █ █ █ model /model
|
| (APIServer pid=1) INFO 10-05 23:03:40 [utils.py:287] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
|
| (APIServer pid=1) INFO 10-05 23:03:40 [utils.py:287] |
| (APIServer pid=1) INFO 10-05 23:03:40 [utils.py:223] non-default args: {'host': '0.0.0.0', 'enable_auto_tool_choice': True, 'tool_call_parser': 'gemma4', 'model': '/model', 'max_model_len': 262144, 'served_model_name': ['google/gemma-4-12B-it'], 'reasoning_parser': 'gemma4', 'block_size': 64, 'enable_prefix_caching': True, 'max_num_batched_tokens': 16384, 'max_num_seqs': 1, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'gemma4_assistant', 'model': '/drafter', 'num_speculative_tokens': 5}, 'additional_config': {'tt': {'trace_mode': 'all', 'enable_model_warmup': True, 'trace_region_size': 200000000}}} |
| (APIServer pid=1) INFO 10-05 23:03:46 [model.py:526] Resolved architecture: Gemma4UnifiedForConditionalGeneration |
| (APIServer pid=1) INFO 10-05 23:03:46 [model.py:1550] Using max model len 262144 |
| (APIServer pid=1) INFO 10-05 23:03:46 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=16384. |
| (APIServer pid=1) WARNING 10-05 23:03:46 [vllm.py:662] Async scheduling not supported with gemma4_assistant-based speculative decoding and will be disabled. |
| (APIServer pid=1) INFO 10-05 23:03:46 [vllm.py:698] Asynchronous scheduling is disabled. |
| (APIServer pid=1) WARNING 10-05 23:03:46 [vllm.py:743] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (APIServer pid=1) INFO 10-05 23:03:47 [platform.py:796] Automatic prefix caching is enabled |
| INFO 10-05 23:03:50 [__init__.py:44] Available plugins for group vllm.platform_plugins: |
| INFO 10-05 23:03:50 [__init__.py:46] - tt -> vllm_tt_plugin.entrypoints:platform_plugin |
| INFO 10-05 23:03:50 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load. |
| INFO 10-05 23:03:51 [__init__.py:212] Platform plugin tt is activated |
| INFO 10-05 23:03:53 [importing.py:44] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors. |
| INFO 10-05 23:03:53 [importing.py:68] Triton not installed or not compatible; certain GPU-related functions will not be available. |
| (EngineCore_DP0 pid=150) INFO 10-05 23:03:54 [core.py:98] Initializing a V1 LLM engine (v0.1.dev14187+g03fa3af2e) with config: model='/model', speculative_config=SpeculativeConfig(method='gemma4_assistant', model='/drafter', num_spec_tokens=5), tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=None, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='gemma4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=google/gemma-4-12B-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'eager', 'custom_ops': ['all'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': None, 'compile_ranges_split_points': [16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []} |
| (EngineCore_DP0 pid=150) INFO 10-05 23:03:54 [platform.py:796] Automatic prefix caching is enabled |
| 2026-10-05 23:03:54.651 | info | Device | Opening user mode device driver (tt_cluster.cpp:228) |
| 2026-10-05 23:03:54.651 | info | UMD | Cluster constructor started. (cluster.cpp:378) |
| 2026-10-05 23:03:54.651 | info | UMD | Creating TopologyDiscovery for architecture: blackhole (topology_discovery.cpp:95) |
| 2026-10-05 23:03:54.651 | info | UMD | Starting topology discovery. (topology_discovery.cpp:114) |
| 2026-10-05 23:03:54.654 | info | UMD | Established firmware bundle version: 19.15.0 (topology_discovery.cpp:574) |
| 2026-10-05 23:03:54.654 | info | UMD | Completed topology discovery. (topology_discovery.cpp:118) |
| 2026-10-05 23:03:54.680 | warning | UMD | hwloc_set_area_membind(): failed for physical_device_id: 0 on NodeSet: [0] with errno: Operation not permitted (pid: 150 tid: 135962864665024) (cpuset_lib.cpp:440) |
| 2026-10-05 23:03:54.680 | warning | UMD | ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 0 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893). (silicon_sysmem_manager.cpp:271) |
| 2026-10-05 23:03:54.680 | info | UMD | Opening local chip ids/PCIe ids: {0}/[0] and remote chip ids {} (cluster.cpp:168) |
| 2026-10-05 23:03:54.680 | info | UMD | IOMMU: disabled (cluster.cpp:142) |
| 2026-10-05 23:03:54.680 | info | UMD | KMD version: 2.11.0 (cluster.cpp:145) |
| 2026-10-05 23:03:54.680 | info | UMD | Cluster constructor completed. (cluster.cpp:574) |
| 2026-10-05 23:03:54.681 | info | UMD | Starting devices in cluster (cluster.cpp:1092) |
| 2026-10-05 23:03:54.727 | info | UMD | Starting devices in cluster completed. (cluster.cpp:1100) |
| (EngineCore_DP0 pid=150) INFO 10-05 23:03:54 [worker.py:781] Attempting to open mesh device with grid shape (1, 1) |
| 2026-10-05 23:03:54.766 | info | Distributed | Using auto discovery to generate mesh graph. (metal_env.cpp:437) |
| 2026-10-05 23:03:54.766 | info | Distributed | Constructing control plane using auto-discovery (no mesh graph descriptor). (metal_env.cpp:499) |
| 2026-10-05 23:03:54.766 | warning | Always | Unknown motherboard 'ROG STRIX B650-A GAMING WIFI' for chip_id=0 (bus_id=0x1) — falling back to bus_id as tray_id. Add this motherboard and its bus IDs to mobo_to_bus_ids in physical_system_discovery.cpp. (physical_system_discovery.cpp:112) |
| 2026-10-05 23:03:54.768 | info | Fabric | Logical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {0:1} (topology_mapper_utils.cpp:2404) |
| 2026-10-05 23:03:54.768 | info | Fabric | Physical multi-mesh adjacency: intermesh degree histogram {0:1}; intra-mesh degree histograms mesh0 {0:1} (topology_mapper_utils.cpp:2412) |
| 2026-10-05 23:03:54.779 | info | BuildKernels | Using pre-compiled firmware from: /home/container_app_user/tt-metal/tt_metal/pre-compiled/15570547042316605318/ (build_env_manager.cpp:347) |
| (EngineCore_DP0 pid=150) INFO 10-05 23:03:54 [worker.py:794] multidevice with 1 devices and grid (1, 1) is created |
| (EngineCore_DP0 pid=150) INFO 10-05 23:03:54 [model_runner.py:182] TTModelRunner: trace_mode=all, sample_on_device_mode=None, enable_model_warmup=True |
| (EngineCore_DP0 pid=150) WARNING 10-05 23:03:54 [__init__.py:204] min_p, logit_bias, and min_tokens parameters won't currently work with speculative decoding enabled. |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:08.647 | INFO | models.demos.gemma4.tt.model_config:load_state_dict:143 - Loading 1 safetensor files from /drafter |
| (EngineCore_DP0 pid=150)
Loading safetensors: 0%| | 0/1 [00:00<?, ?it/s]
Loading safetensors: 100%|██████████| 1/1 [00:00<00:00, 1403.72it/s] |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:08.648 | INFO | models.demos.gemma4.tt.model_config:load_state_dict:153 - Loaded 48 tensors |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:08.703 | INFO | models.demos.gemma4.tt.serving_vllm:initialize_vllm_model:159 - gemma-4 single-user model loaded in 6.0s (drafter K=5) |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:08 [worker.py:546] Getting max_tokens_all_users=262144 for number of blocks in KV cache from generator '<class 'models.demos.gemma4.tt.serving_vllm.Gemma4SingleUserForCausalLM'>'. |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:08 [kv_cache_utils.py:820] Overriding num_gpu_blocks=837723144972 with num_gpu_blocks_override=4097 |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:08 [kv_cache_utils.py:1308] GPU KV cache size: 262,208 tokens |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:08 [kv_cache_utils.py:1313] Maximum concurrency for 262,144 tokens per request: 1.00x |
| 2026-10-05 23:04:08.731 | info | Metal | Finalize dfb: dfb_offset == base_offset: 80, dfb size: 32, return value: 112 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:08.731 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:08.731 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:08.731 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:09.032 | info | Metal | Finalize dfb: dfb_offset == base_offset: 80, dfb size: 32, return value: 112 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:09.032 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:09.032 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:09.032 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:12.142 | INFO | models.demos.gemma4.tt.serving_vllm:allocate_kv_cache_per_layer:212 - KV: full 4097 blocks x 64, sliding ring 18432 tokens; DRAM allocated 26.96 GB |
| 2026-10-05 23:04:12.200 | info | Metal | Finalize dfb: dfb_offset == base_offset: 64, dfb size: 16, return value: 80 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:12.200 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:12.200 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:12.200 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:12.382 | info | Metal | Finalize dfb: dfb_offset == base_offset: 64, dfb size: 16, return value: 80 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:12.382 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:12.382 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:12.382 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:43 [prefix_resume.py:79] Prefix resume: chunk 16384, block 64, max snapshots 4 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.095 | INFO | models.demos.gemma4.tt.serving_vllm:warmup_model_prefill:241 - prefill warmup 31.0s |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.095 | INFO | models.demos.gemma4.tt.single_user:_capture:254 - decode trace (re)capture at pos 16: ['LogitsDecoder', 'SpecGreedyDecoder'] |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.279 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.279 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.314 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.315 | INFO | tracy:signpost:49 - gemma4_lm_head |
| 2026-10-05 23:04:43.345 | warning | Metal | Allocating device buffers is unsafe due to the existence of an active trace. These buffers may be corrupted once a trace is executed. (allocator.cpp:123) |
| 2026-10-05 23:04:43.506 | info | Metal | Finalize dfb: dfb_offset == base_offset: 64, dfb size: 16, return value: 80 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.506 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.506 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.506 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.506 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.507 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.576 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.576 | INFO | tracy:signpost:49 - gemma4_lm_head |
| 2026-10-05 23:04:43.801 | info | Metal | Finalize dfb: dfb_offset == base_offset: 64, dfb size: 16, return value: 80 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.801 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.801 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.801 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.801 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.802 | INFO | tracy:signpost:49 - gemma4_lm_head |
| 2026-10-05 23:04:43.901 | info | Metal | Finalize dfb: dfb_offset == base_offset: 64, dfb size: 16, return value: 80 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.901 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.901 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.901 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.901 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.902 | INFO | tracy:signpost:49 - gemma4_lm_head |
| 2026-10-05 23:04:43.996 | info | Metal | Finalize dfb: dfb_offset == base_offset: 64, dfb size: 16, return value: 80 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.996 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.996 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:43.996 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.996 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:43.997 | INFO | tracy:signpost:49 - gemma4_lm_head |
| 2026-10-05 23:04:44.099 | info | Metal | Finalize dfb: dfb_offset == base_offset: 64, dfb size: 16, return value: 80 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:44.099 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:44.099 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| 2026-10-05 23:04:44.099 | info | Metal | Finalize dfb: dfb_offset == base_offset: 0, dfb size: 0, return value: 0 (dataflow_buffer.cpp:1656) |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.099 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.100 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.161 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.161 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.235 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.236 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.306 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.306 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.382 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.383 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.460 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.461 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.534 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.535 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.622 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.623 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.661 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.661 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.746 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.747 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.803 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.803 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.914 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.914 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.957 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:44.957 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.001 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.001 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.045 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.045 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.089 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.089 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.158 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.158 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.233 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.234 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.312 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.312 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.391 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.391 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.471 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.472 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:45.507 | INFO | models.demos.gemma4.tt.serving_vllm:warmup_model_decode:250 - decode trace capture 2.4s |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:45 [core.py:286] init engine (profile, create kv cache, warmup model) took 36.80 seconds |
| (EngineCore_DP0 pid=150) WARNING 10-05 23:04:47 [scheduler.py:166] Using custom scheduler class vllm_tt_plugin.scheduler.TTScheduler. This scheduler interface is not public and compatibility may not be maintained. |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:47 [core.py:126] Scheduler class: TTScheduler, async_scheduling=False |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:47 [prefix_resume.py:253] Prefix resume attached to the KV cache manager |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:48 [vllm.py:698] Asynchronous scheduling is disabled. |
| (EngineCore_DP0 pid=150) WARNING 10-05 23:04:48 [vllm.py:743] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. |
| (EngineCore_DP0 pid=150) INFO 10-05 23:04:48 [platform.py:796] Automatic prefix caching is enabled |
| (APIServer pid=1) INFO 10-05 23:04:48 [api_server.py:495] Supported tasks: ['generate'] |
| (APIServer pid=1) INFO 10-05 23:04:48 [parser_manager.py:202] "auto" tool choice has been enabled. |
| (APIServer pid=1) WARNING 10-05 23:04:48 [model.py:1351] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 64, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`. |
| (APIServer pid=1) INFO 10-05 23:04:48 [parser_manager.py:202] "auto" tool choice has been enabled. |
| (APIServer pid=1) INFO 10-05 23:04:48 [serving.py:185] Warming up chat template processing... |
| (APIServer pid=1) INFO 10-05 23:04:48 [hf.py:318] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. |
| (APIServer pid=1) INFO 10-05 23:04:48 [serving.py:210] Chat template warmup completed in 71.5ms |
| (APIServer pid=1) INFO 10-05 23:04:48 [parser_manager.py:202] "auto" tool choice has been enabled. |
| (APIServer pid=1) INFO 10-05 23:04:48 [api_server.py:500] Starting vLLM API server 0 on http: |
| (APIServer pid=1) INFO 10-05 23:04:48 [launcher.py:38] Available routes are: |
| (APIServer pid=1) INFO 10-05 23:04:48 [launcher.py:47] Route: /openapi.json, Methods: HEAD, GET |
| (APIServer pid=1) INFO 10-05 23:04:48 [launcher.py:47] Route: /docs, Methods: HEAD, GET |
| (APIServer pid=1) INFO 10-05 23:04:48 [launcher.py:47] Route: /docs/oauth2-redirect, Methods: HEAD, GET |
| (APIServer pid=1) INFO 10-05 23:04:48 [launcher.py:47] Route: /redoc, Methods: HEAD, GET |
| (APIServer pid=1) INFO 10-05 23:04:48 [launcher.py:47] Route: /metrics, Methods: GET |
| (APIServer pid=1) INFO: Started server process [1] |
| (APIServer pid=1) INFO: Waiting for application startup. |
| (APIServer pid=1) INFO: Application startup complete. |
| (APIServer pid=1) INFO: 172.17.0.1:38642 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:51.917 | INFO | models.demos.gemma4.tt.single_user:_capture:254 - decode trace (re)capture at pos 65: ['LogitsDecoder', 'SpecGreedyDecoder'] |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:51.971 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:51.971 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.011 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.011 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.100 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.101 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.158 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.158 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.269 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.269 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.313 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.313 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.356 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.357 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.400 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.400 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.444 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.445 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.516 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.516 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.593 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.593 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.672 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.673 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.753 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.754 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.834 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:52.834 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:53.640 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=138 last 64 steps: tokens/step=4.59 drafts/step=5.00 step_ms=66.0 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:38658 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:04:57.048 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=307 last 64 steps: tokens/step=4.67 drafts/step=5.00 step_ms=51.1 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:38670 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:04:59 [loggers.py:259] Engine 000: Avg prompt throughput: 31.8 tokens/s, Avg generation throughput: 73.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:04:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.70, Accepted throughput: 57.62 tokens/s, Drafted throughput: 77.93 tokens/s, Accepted: 610 tokens, Drafted: 825 tokens, Per-position acceptance rate: 0.915, 0.824, 0.752, 0.667, 0.539, Avg Draft acceptance rate: 73.9% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:00.468 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=297 last 64 steps: tokens/step=5.14 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:55766 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:00.963 | INFO | models.demos.gemma4.tt.single_user:_capture:254 - decode trace (re)capture at pos 150: ['LogitsDecoder', 'SpecGreedyDecoder'] |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.016 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.017 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.056 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.056 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.143 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.143 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.200 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.201 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.311 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.311 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.354 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.354 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.396 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.397 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.440 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.440 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.483 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.483 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.553 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.553 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.627 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.628 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.705 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.705 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.784 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.784 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.863 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:01.863 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (APIServer pid=1) INFO: 172.17.0.1:55776 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:05.116 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=113 last 64 steps: tokens/step=4.42 drafts/step=5.00 step_ms=66.1 host_gap_ms=6.5 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:08.432 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=392 last 64 steps: tokens/step=4.34 drafts/step=5.00 step_ms=51.4 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:55788 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:05:09 [loggers.py:259] Engine 000: Avg prompt throughput: 32.4 tokens/s, Avg generation throughput: 74.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:05:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.46, Accepted throughput: 57.50 tokens/s, Drafted throughput: 83.00 tokens/s, Accepted: 575 tokens, Drafted: 830 tokens, Per-position acceptance rate: 0.898, 0.795, 0.663, 0.608, 0.500, Avg Draft acceptance rate: 69.3% |
| (APIServer pid=1) INFO: 172.17.0.1:44520 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:11.938 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=160 last 64 steps: tokens/step=4.92 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.7 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:15.280 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=452 last 64 steps: tokens/step=4.58 drafts/step=5.00 step_ms=51.8 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:44536 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:18.900 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=435 last 64 steps: tokens/step=4.69 drafts/step=5.00 step_ms=51.7 host_gap_ms=4.9 |
| (APIServer pid=1) INFO 10-05 23:05:19 [loggers.py:259] Engine 000: Avg prompt throughput: 25.2 tokens/s, Avg generation throughput: 87.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:05:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.72, Accepted throughput: 68.79 tokens/s, Drafted throughput: 92.49 tokens/s, Accepted: 688 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.908, 0.800, 0.741, 0.654, 0.616, Avg Draft acceptance rate: 74.4% |
| (APIServer pid=1) INFO: 172.17.0.1:53458 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:22.570 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=202 last 64 steps: tokens/step=5.02 drafts/step=5.00 step_ms=52.3 host_gap_ms=5.1 |
| (APIServer pid=1) INFO: 172.17.0.1:53464 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:25.994 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=230 last 64 steps: tokens/step=4.84 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:53468 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:05:29 [loggers.py:259] Engine 000: Avg prompt throughput: 29.9 tokens/s, Avg generation throughput: 90.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:05:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.83, Accepted throughput: 71.69 tokens/s, Drafted throughput: 93.49 tokens/s, Accepted: 717 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.904, 0.840, 0.765, 0.706, 0.620, Avg Draft acceptance rate: 76.7% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:29.423 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=241 last 64 steps: tokens/step=5.22 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48586 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:32.847 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=266 last 64 steps: tokens/step=4.94 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48592 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:36.315 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=123 last 64 steps: tokens/step=3.89 drafts/step=5.00 step_ms=52.1 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:05:39 [loggers.py:259] Engine 000: Avg prompt throughput: 20.4 tokens/s, Avg generation throughput: 78.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:05:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.19, Accepted throughput: 59.90 tokens/s, Drafted throughput: 93.99 tokens/s, Accepted: 599 tokens, Drafted: 940 tokens, Per-position acceptance rate: 0.851, 0.723, 0.617, 0.532, 0.463, Avg Draft acceptance rate: 63.7% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:39.625 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=356 last 64 steps: tokens/step=3.58 drafts/step=5.00 step_ms=51.3 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:54378 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:43.082 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=232 last 64 steps: tokens/step=5.08 drafts/step=5.00 step_ms=51.9 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:54392 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:46.688 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=272 last 64 steps: tokens/step=5.08 drafts/step=5.00 step_ms=51.5 host_gap_ms=4.8 |
| (APIServer pid=1) INFO: 172.17.0.1:54396 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:05:49 [loggers.py:259] Engine 000: Avg prompt throughput: 31.8 tokens/s, Avg generation throughput: 92.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:05:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 5.04, Accepted throughput: 73.50 tokens/s, Drafted throughput: 91.00 tokens/s, Accepted: 735 tokens, Drafted: 910 tokens, Per-position acceptance rate: 0.956, 0.890, 0.819, 0.753, 0.621, Avg Draft acceptance rate: 80.8% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:50.114 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=267 last 64 steps: tokens/step=5.19 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48036 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:48040 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:53.634 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=99 last 64 steps: tokens/step=5.06 drafts/step=5.00 step_ms=51.3 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:48046 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:05:57.034 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=223 last 64 steps: tokens/step=4.39 drafts/step=5.00 step_ms=51.0 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:05:59 [loggers.py:259] Engine 000: Avg prompt throughput: 26.8 tokens/s, Avg generation throughput: 91.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:05:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.86, Accepted throughput: 72.10 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 721 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.930, 0.861, 0.765, 0.711, 0.588, Avg Draft acceptance rate: 77.1% |
| (APIServer pid=1) INFO: 172.17.0.1:58204 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:00.472 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=205 last 64 steps: tokens/step=5.23 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.1 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:03.824 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=486 last 64 steps: tokens/step=4.31 drafts/step=5.00 step_ms=52.0 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:58214 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:07.239 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=331 last 64 steps: tokens/step=4.28 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:58222 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:06:09 [loggers.py:259] Engine 000: Avg prompt throughput: 28.2 tokens/s, Avg generation throughput: 83.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:06:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.45, Accepted throughput: 64.60 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 646 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.861, 0.791, 0.668, 0.599, 0.535, Avg Draft acceptance rate: 69.1% |
| (APIServer pid=1) INFO: 172.17.0.1:46902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:10.745 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=155 last 64 steps: tokens/step=4.84 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:46906 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:14.152 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=314 last 64 steps: tokens/step=4.86 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:46916 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:17.606 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=216 last 64 steps: tokens/step=4.95 drafts/step=5.00 step_ms=51.9 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:06:19 [loggers.py:259] Engine 000: Avg prompt throughput: 26.3 tokens/s, Avg generation throughput: 91.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:06:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.86, Accepted throughput: 72.20 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 722 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.920, 0.850, 0.781, 0.690, 0.620, Avg Draft acceptance rate: 77.2% |
| (APIServer pid=1) INFO: 172.17.0.1:36478 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:21.037 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=217 last 64 steps: tokens/step=5.03 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:36484 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:24.469 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=204 last 64 steps: tokens/step=5.09 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:36486 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:27.886 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=305 last 64 steps: tokens/step=4.94 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:53342 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:06:29 [loggers.py:259] Engine 000: Avg prompt throughput: 39.9 tokens/s, Avg generation throughput: 91.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:06:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.96, Accepted throughput: 72.80 tokens/s, Drafted throughput: 92.00 tokens/s, Accepted: 728 tokens, Drafted: 920 tokens, Per-position acceptance rate: 0.935, 0.870, 0.788, 0.707, 0.658, Avg Draft acceptance rate: 79.1% |
| (APIServer pid=1) INFO: 172.17.0.1:53352 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:31.396 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=146 last 64 steps: tokens/step=4.67 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:53356 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:34.807 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=195 last 64 steps: tokens/step=4.55 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:53360 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:38.232 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=215 last 64 steps: tokens/step=5.41 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:43074 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:06:39 [loggers.py:259] Engine 000: Avg prompt throughput: 33.1 tokens/s, Avg generation throughput: 91.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:06:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.89, Accepted throughput: 72.40 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 724 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.952, 0.871, 0.769, 0.694, 0.608, Avg Draft acceptance rate: 77.8% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:41.652 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=399 last 64 steps: tokens/step=5.61 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:43088 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:43094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:45.166 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=185 last 64 steps: tokens/step=5.20 drafts/step=5.00 step_ms=51.2 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:43098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:48.576 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=282 last 64 steps: tokens/step=4.94 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:06:49 [loggers.py:259] Engine 000: Avg prompt throughput: 25.2 tokens/s, Avg generation throughput: 97.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:06:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 5.18, Accepted throughput: 78.20 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 782 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.963, 0.914, 0.866, 0.759, 0.679, Avg Draft acceptance rate: 83.6% |
| (APIServer pid=1) INFO: 172.17.0.1:35366 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:51.999 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=296 last 64 steps: tokens/step=4.72 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:55.378 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=553 last 64 steps: tokens/step=4.03 drafts/step=5.00 step_ms=52.4 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:35374 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:06:58.806 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=375 last 64 steps: tokens/step=4.88 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:37012 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:06:59 [loggers.py:259] Engine 000: Avg prompt throughput: 29.6 tokens/s, Avg generation throughput: 82.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:06:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.42, Accepted throughput: 63.60 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 636 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.887, 0.780, 0.672, 0.581, 0.500, Avg Draft acceptance rate: 68.4% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:02.233 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=400 last 64 steps: tokens/step=4.58 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:37024 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:37038 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:05.794 | INFO | models.demos.gemma4.tt.single_user:_capture:254 - decode trace (re)capture at pos 164: ['LogitsDecoder', 'SpecGreedyDecoder'] |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:05.850 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:05.851 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:05.890 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:05.891 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:05.981 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:05.981 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.039 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.039 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.147 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.148 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.191 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.191 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.233 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.234 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.277 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.277 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.320 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.321 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.391 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.391 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.466 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.466 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.544 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.544 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.625 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.626 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.708 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.708 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:06.899 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=172 last 64 steps: tokens/step=4.88 drafts/step=5.00 step_ms=66.4 host_gap_ms=6.5 |
| (APIServer pid=1) INFO 10-05 23:07:09 [loggers.py:259] Engine 000: Avg prompt throughput: 25.1 tokens/s, Avg generation throughput: 76.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:07:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.57, Accepted throughput: 59.30 tokens/s, Drafted throughput: 83.00 tokens/s, Accepted: 593 tokens, Drafted: 830 tokens, Per-position acceptance rate: 0.934, 0.831, 0.699, 0.596, 0.512, Avg Draft acceptance rate: 71.4% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:10.237 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=409 last 64 steps: tokens/step=3.66 drafts/step=5.00 step_ms=51.7 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:47086 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:13.742 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=159 last 64 steps: tokens/step=4.17 drafts/step=5.00 step_ms=52.7 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:47094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:17.150 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=230 last 64 steps: tokens/step=4.38 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:07:19 [loggers.py:259] Engine 000: Avg prompt throughput: 22.0 tokens/s, Avg generation throughput: 78.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:07:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.20, Accepted throughput: 59.80 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 598 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.888, 0.733, 0.604, 0.529, 0.444, Avg Draft acceptance rate: 64.0% |
| (APIServer pid=1) INFO: 172.17.0.1:42888 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:20.593 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=177 last 64 steps: tokens/step=4.59 drafts/step=5.00 step_ms=51.7 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:42898 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:24.202 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=220 last 64 steps: tokens/step=4.92 drafts/step=5.00 step_ms=51.5 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:42910 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:27.817 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=243 last 64 steps: tokens/step=5.05 drafts/step=5.00 step_ms=51.6 host_gap_ms=4.8 |
| (APIServer pid=1) INFO 10-05 23:07:29 [loggers.py:259] Engine 000: Avg prompt throughput: 37.8 tokens/s, Avg generation throughput: 88.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:07:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.90, Accepted throughput: 70.20 tokens/s, Drafted throughput: 90.00 tokens/s, Accepted: 702 tokens, Drafted: 900 tokens, Per-position acceptance rate: 0.950, 0.872, 0.772, 0.717, 0.589, Avg Draft acceptance rate: 78.0% |
| (APIServer pid=1) INFO: 172.17.0.1:39178 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:31.275 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=119 last 64 steps: tokens/step=4.62 drafts/step=5.00 step_ms=52.0 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:39182 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:34.712 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=84 last 64 steps: tokens/step=5.06 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:39198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:38.114 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=211 last 64 steps: tokens/step=4.92 drafts/step=5.00 step_ms=51.0 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:07:39 [loggers.py:259] Engine 000: Avg prompt throughput: 24.6 tokens/s, Avg generation throughput: 90.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:07:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.84, Accepted throughput: 71.50 tokens/s, Drafted throughput: 92.99 tokens/s, Accepted: 715 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.919, 0.833, 0.785, 0.694, 0.613, Avg Draft acceptance rate: 76.9% |
| (APIServer pid=1) INFO: 172.17.0.1:55046 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:41.522 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=288 last 64 steps: tokens/step=5.02 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:55062 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:55078 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:45.045 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=120 last 64 steps: tokens/step=4.59 drafts/step=5.00 step_ms=51.3 host_gap_ms=3.8 |
| (APIServer pid=1) INFO: 172.17.0.1:55094 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:48.641 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=172 last 64 steps: tokens/step=4.67 drafts/step=5.00 step_ms=51.3 host_gap_ms=4.9 |
| (APIServer pid=1) INFO 10-05 23:07:49 [loggers.py:259] Engine 000: Avg prompt throughput: 40.6 tokens/s, Avg generation throughput: 84.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:07:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.64, Accepted throughput: 66.20 tokens/s, Drafted throughput: 91.00 tokens/s, Accepted: 662 tokens, Drafted: 910 tokens, Per-position acceptance rate: 0.901, 0.813, 0.731, 0.637, 0.555, Avg Draft acceptance rate: 72.7% |
| (APIServer pid=1) INFO: 172.17.0.1:37296 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:52.061 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=187 last 64 steps: tokens/step=4.97 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:37298 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:55.474 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=177 last 64 steps: tokens/step=4.64 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:37310 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:40136 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:07:58.979 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=169 last 64 steps: tokens/step=5.11 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.7 |
| (APIServer pid=1) INFO 10-05 23:07:59 [loggers.py:259] Engine 000: Avg prompt throughput: 40.4 tokens/s, Avg generation throughput: 91.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:07:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.90, Accepted throughput: 72.21 tokens/s, Drafted throughput: 92.51 tokens/s, Accepted: 722 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.930, 0.854, 0.778, 0.719, 0.622, Avg Draft acceptance rate: 78.1% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:02.333 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=496 last 64 steps: tokens/step=5.11 drafts/step=5.00 step_ms=52.0 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:40142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:05.779 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=388 last 64 steps: tokens/step=4.83 drafts/step=5.00 step_ms=51.8 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:40158 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:60218 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:08:09 [loggers.py:259] Engine 000: Avg prompt throughput: 26.9 tokens/s, Avg generation throughput: 92.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:08:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.93, Accepted throughput: 73.49 tokens/s, Drafted throughput: 93.49 tokens/s, Accepted: 735 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.947, 0.861, 0.781, 0.722, 0.620, Avg Draft acceptance rate: 78.6% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:09.273 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=243 last 64 steps: tokens/step=4.84 drafts/step=5.00 step_ms=50.9 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:60222 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:60236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:12.776 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=171 last 64 steps: tokens/step=5.03 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.6 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:16.129 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=457 last 64 steps: tokens/step=4.41 drafts/step=5.00 step_ms=51.8 host_gap_ms=0.5 |
| (APIServer pid=1) INFO: 172.17.0.1:60242 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:08:19 [loggers.py:259] Engine 000: Avg prompt throughput: 31.3 tokens/s, Avg generation throughput: 81.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:08:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.38, Accepted throughput: 62.90 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 629 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.849, 0.769, 0.688, 0.591, 0.484, Avg Draft acceptance rate: 67.6% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:19.803 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=210 last 64 steps: tokens/step=4.09 drafts/step=5.00 step_ms=52.3 host_gap_ms=5.1 |
| (APIServer pid=1) INFO: 172.17.0.1:49336 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:23.424 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=207 last 64 steps: tokens/step=4.69 drafts/step=5.00 step_ms=51.7 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:49342 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:26.851 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=184 last 64 steps: tokens/step=4.88 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:56536 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:08:29 [loggers.py:259] Engine 000: Avg prompt throughput: 31.5 tokens/s, Avg generation throughput: 87.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:08:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.74, Accepted throughput: 68.40 tokens/s, Drafted throughput: 91.50 tokens/s, Accepted: 684 tokens, Drafted: 915 tokens, Per-position acceptance rate: 0.902, 0.814, 0.749, 0.678, 0.596, Avg Draft acceptance rate: 74.8% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:30.266 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=223 last 64 steps: tokens/step=4.77 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:56550 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:33.709 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=181 last 64 steps: tokens/step=5.02 drafts/step=5.00 step_ms=51.7 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:56554 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:37.129 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=209 last 64 steps: tokens/step=5.25 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:58236 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:08:39 [loggers.py:259] Engine 000: Avg prompt throughput: 25.9 tokens/s, Avg generation throughput: 91.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:08:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.90, Accepted throughput: 72.89 tokens/s, Drafted throughput: 93.49 tokens/s, Accepted: 729 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.941, 0.850, 0.775, 0.695, 0.636, Avg Draft acceptance rate: 78.0% |
| (APIServer pid=1) INFO: 172.17.0.1:58244 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:40.646 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=100 last 64 steps: tokens/step=4.45 drafts/step=5.00 step_ms=51.2 host_gap_ms=3.7 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:43.969 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=409 last 64 steps: tokens/step=4.83 drafts/step=5.00 step_ms=51.5 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:47.479 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=161 last 64 steps: tokens/step=4.77 drafts/step=5.00 step_ms=51.2 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:38588 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:08:49 [loggers.py:259] Engine 000: Avg prompt throughput: 34.3 tokens/s, Avg generation throughput: 88.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:08:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.75, Accepted throughput: 69.40 tokens/s, Drafted throughput: 92.50 tokens/s, Accepted: 694 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.930, 0.811, 0.741, 0.665, 0.605, Avg Draft acceptance rate: 75.0% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:50.890 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=279 last 64 steps: tokens/step=5.03 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:38598 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:54.510 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=407 last 64 steps: tokens/step=4.83 drafts/step=5.00 step_ms=51.7 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:38608 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:08:57.958 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=275 last 64 steps: tokens/step=4.78 drafts/step=5.00 step_ms=51.7 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:08:59 [loggers.py:259] Engine 000: Avg prompt throughput: 26.7 tokens/s, Avg generation throughput: 91.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:08:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.92, Accepted throughput: 72.50 tokens/s, Drafted throughput: 92.50 tokens/s, Accepted: 725 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.935, 0.854, 0.778, 0.719, 0.632, Avg Draft acceptance rate: 78.4% |
| (APIServer pid=1) INFO: 172.17.0.1:48660 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:01.408 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=254 last 64 steps: tokens/step=5.30 drafts/step=5.00 step_ms=51.8 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48666 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:04.881 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=101 last 64 steps: tokens/step=4.69 drafts/step=5.00 step_ms=52.2 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48678 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:08.282 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=128 last 64 steps: tokens/step=4.50 drafts/step=5.00 step_ms=51.1 host_gap_ms=2.0 |
| (APIServer pid=1) INFO 10-05 23:09:09 [loggers.py:259] Engine 000: Avg prompt throughput: 28.9 tokens/s, Avg generation throughput: 87.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:09:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.69, Accepted throughput: 68.70 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 687 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.882, 0.817, 0.737, 0.677, 0.581, Avg Draft acceptance rate: 73.9% |
| (APIServer pid=1) INFO: 172.17.0.1:45894 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:11.699 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=133 last 64 steps: tokens/step=4.95 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:45898 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:15.095 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=274 last 64 steps: tokens/step=4.42 drafts/step=5.00 step_ms=51.0 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:45902 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:56348 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:18.590 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=176 last 64 steps: tokens/step=5.00 drafts/step=5.00 step_ms=51.0 host_gap_ms=3.6 |
| (APIServer pid=1) INFO 10-05 23:09:19 [loggers.py:259] Engine 000: Avg prompt throughput: 32.8 tokens/s, Avg generation throughput: 87.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:09:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.70, Accepted throughput: 68.90 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 689 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.914, 0.817, 0.726, 0.661, 0.586, Avg Draft acceptance rate: 74.1% |
| (APIServer pid=1) INFO: 172.17.0.1:56350 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:56364 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.418 | INFO | models.demos.gemma4.tt.single_user:_capture:254 - decode trace (re)capture at pos 63: ['LogitsDecoder', 'SpecGreedyDecoder'] |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.473 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.473 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.512 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.513 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.598 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.599 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.654 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.655 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.761 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.761 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.803 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.803 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.845 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.845 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.887 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.887 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.929 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.929 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.997 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:21.997 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:22.071 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:22.071 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:22.146 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:22.146 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:22.226 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:22.226 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:22.310 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:22.310 | INFO | tracy:signpost:49 - gemma4_lm_head |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:23.009 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=122 last 64 steps: tokens/step=5.06 drafts/step=5.00 step_ms=65.4 host_gap_ms=3.6 |
| (APIServer pid=1) INFO: 172.17.0.1:56380 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:26.435 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=170 last 64 steps: tokens/step=5.14 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:09:29 [loggers.py:259] Engine 000: Avg prompt throughput: 25.5 tokens/s, Avg generation throughput: 78.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:09:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.60, Accepted throughput: 60.80 tokens/s, Drafted throughput: 84.50 tokens/s, Accepted: 608 tokens, Drafted: 845 tokens, Per-position acceptance rate: 0.888, 0.805, 0.746, 0.621, 0.538, Avg Draft acceptance rate: 72.0% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:29.768 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=405 last 64 steps: tokens/step=3.73 drafts/step=5.00 step_ms=51.6 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:55068 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:33.270 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=161 last 64 steps: tokens/step=4.08 drafts/step=5.00 step_ms=52.6 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:55072 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:36.678 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=227 last 64 steps: tokens/step=4.11 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.0 |
| (APIServer pid=1) INFO 10-05 23:09:39 [loggers.py:259] Engine 000: Avg prompt throughput: 24.5 tokens/s, Avg generation throughput: 75.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:09:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 3.99, Accepted throughput: 56.30 tokens/s, Drafted throughput: 94.00 tokens/s, Accepted: 563 tokens, Drafted: 940 tokens, Per-position acceptance rate: 0.862, 0.686, 0.585, 0.489, 0.372, Avg Draft acceptance rate: 59.9% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:40.029 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=463 last 64 steps: tokens/step=3.69 drafts/step=5.00 step_ms=52.0 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:33564 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:43.458 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=322 last 64 steps: tokens/step=5.28 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:33572 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:46.874 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=321 last 64 steps: tokens/step=4.39 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:09:49 [loggers.py:259] Engine 000: Avg prompt throughput: 16.8 tokens/s, Avg generation throughput: 83.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:09:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.43, Accepted throughput: 64.20 tokens/s, Drafted throughput: 93.49 tokens/s, Accepted: 642 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.888, 0.759, 0.663, 0.599, 0.524, Avg Draft acceptance rate: 68.7% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:50.254 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=552 last 64 steps: tokens/step=3.61 drafts/step=5.00 step_ms=52.5 host_gap_ms=0.3 |
| (APIServer pid=1) INFO: 172.17.0.1:33574 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:60188 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:53.781 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=117 last 64 steps: tokens/step=5.16 drafts/step=5.00 step_ms=51.4 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:60196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:09:57.178 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=223 last 64 steps: tokens/step=4.91 drafts/step=5.00 step_ms=51.1 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:60190 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:09:59 [loggers.py:259] Engine 000: Avg prompt throughput: 41.5 tokens/s, Avg generation throughput: 89.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:09:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.84, Accepted throughput: 71.00 tokens/s, Drafted throughput: 92.50 tokens/s, Accepted: 710 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.930, 0.822, 0.757, 0.686, 0.643, Avg Draft acceptance rate: 76.8% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:00.607 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=310 last 64 steps: tokens/step=4.69 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:03.991 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=577 last 64 steps: tokens/step=4.25 drafts/step=5.00 step_ms=52.5 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:60192 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:07.587 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=349 last 64 steps: tokens/step=4.94 drafts/step=5.00 step_ms=51.7 host_gap_ms=4.4 |
| (APIServer pid=1) INFO: 172.17.0.1:42308 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:10:09 [loggers.py:259] Engine 000: Avg prompt throughput: 17.1 tokens/s, Avg generation throughput: 85.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:10:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.53, Accepted throughput: 66.70 tokens/s, Drafted throughput: 94.50 tokens/s, Accepted: 667 tokens, Drafted: 945 tokens, Per-position acceptance rate: 0.868, 0.778, 0.688, 0.630, 0.566, Avg Draft acceptance rate: 70.6% |
| (APIServer pid=1) INFO: 172.17.0.1:42324 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:11.109 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=76 last 64 steps: tokens/step=5.50 drafts/step=5.00 step_ms=51.4 host_gap_ms=3.6 |
| (APIServer pid=1) INFO: 172.17.0.1:42328 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:14.505 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=184 last 64 steps: tokens/step=4.86 drafts/step=5.00 step_ms=51.0 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:42332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:18.109 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=300 last 64 steps: tokens/step=4.91 drafts/step=5.00 step_ms=51.4 host_gap_ms=4.9 |
| (APIServer pid=1) INFO 10-05 23:10:19 [loggers.py:259] Engine 000: Avg prompt throughput: 29.9 tokens/s, Avg generation throughput: 93.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:10:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 5.07, Accepted throughput: 74.40 tokens/s, Drafted throughput: 91.50 tokens/s, Accepted: 744 tokens, Drafted: 915 tokens, Per-position acceptance rate: 0.940, 0.880, 0.842, 0.732, 0.672, Avg Draft acceptance rate: 81.3% |
| (APIServer pid=1) INFO: 172.17.0.1:39432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:21.541 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=279 last 64 steps: tokens/step=4.88 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:39440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:24.998 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=298 last 64 steps: tokens/step=4.72 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.6 |
| (APIServer pid=1) INFO: 172.17.0.1:39450 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:28.491 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=199 last 64 steps: tokens/step=4.39 drafts/step=5.00 step_ms=50.9 host_gap_ms=3.7 |
| (APIServer pid=1) INFO 10-05 23:10:29 [loggers.py:259] Engine 000: Avg prompt throughput: 38.0 tokens/s, Avg generation throughput: 84.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:10:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.54, Accepted throughput: 65.40 tokens/s, Drafted throughput: 92.50 tokens/s, Accepted: 654 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.876, 0.773, 0.692, 0.627, 0.568, Avg Draft acceptance rate: 70.7% |
| (APIServer pid=1) INFO: 172.17.0.1:36274 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:31.911 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=203 last 64 steps: tokens/step=5.12 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:36280 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:35.517 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=350 last 64 steps: tokens/step=4.95 drafts/step=5.00 step_ms=51.5 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:36284 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:38.930 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=304 last 64 steps: tokens/step=4.17 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.0 |
| (APIServer pid=1) INFO 10-05 23:10:39 [loggers.py:259] Engine 000: Avg prompt throughput: 31.6 tokens/s, Avg generation throughput: 86.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:10:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.72, Accepted throughput: 68.00 tokens/s, Drafted throughput: 91.50 tokens/s, Accepted: 680 tokens, Drafted: 915 tokens, Per-position acceptance rate: 0.902, 0.820, 0.765, 0.667, 0.563, Avg Draft acceptance rate: 74.3% |
| (APIServer pid=1) INFO: 172.17.0.1:50324 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:50330 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:42.448 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=122 last 64 steps: tokens/step=5.06 drafts/step=5.00 step_ms=51.3 host_gap_ms=3.7 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:45.765 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=370 last 64 steps: tokens/step=3.94 drafts/step=5.00 step_ms=51.4 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:50340 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:10:49 [loggers.py:259] Engine 000: Avg prompt throughput: 33.3 tokens/s, Avg generation throughput: 81.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:10:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.39, Accepted throughput: 62.30 tokens/s, Drafted throughput: 92.00 tokens/s, Accepted: 623 tokens, Drafted: 920 tokens, Per-position acceptance rate: 0.842, 0.745, 0.674, 0.614, 0.511, Avg Draft acceptance rate: 67.7% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:49.387 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=334 last 64 steps: tokens/step=4.23 drafts/step=5.00 step_ms=51.7 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:49228 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:52.846 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=178 last 64 steps: tokens/step=4.84 drafts/step=5.00 step_ms=52.0 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:49244 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:56.272 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=240 last 64 steps: tokens/step=5.22 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:49246 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:10:59 [loggers.py:259] Engine 000: Avg prompt throughput: 29.9 tokens/s, Avg generation throughput: 94.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:10:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 5.04, Accepted throughput: 75.10 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 751 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.952, 0.892, 0.812, 0.715, 0.667, Avg Draft acceptance rate: 80.8% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:10:59.706 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=271 last 64 steps: tokens/step=5.14 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:35238 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:03.144 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=204 last 64 steps: tokens/step=4.84 drafts/step=5.00 step_ms=51.7 host_gap_ms=2.0 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:06.493 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=466 last 64 steps: tokens/step=4.08 drafts/step=5.00 step_ms=52.0 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:35244 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:11:09 [loggers.py:259] Engine 000: Avg prompt throughput: 20.4 tokens/s, Avg generation throughput: 78.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:11:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.23, Accepted throughput: 60.10 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 601 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.866, 0.720, 0.608, 0.543, 0.495, Avg Draft acceptance rate: 64.6% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:09.964 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=186 last 64 steps: tokens/step=3.44 drafts/step=5.00 step_ms=52.2 host_gap_ms=2.1 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:13.301 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=440 last 64 steps: tokens/step=3.98 drafts/step=5.00 step_ms=51.8 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:41444 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:16.734 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=319 last 64 steps: tokens/step=4.75 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:41452 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:11:19 [loggers.py:259] Engine 000: Avg prompt throughput: 22.6 tokens/s, Avg generation throughput: 78.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:11:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.21, Accepted throughput: 59.29 tokens/s, Drafted throughput: 92.49 tokens/s, Accepted: 593 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.816, 0.703, 0.611, 0.551, 0.524, Avg Draft acceptance rate: 64.1% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:20.350 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=333 last 64 steps: tokens/step=4.95 drafts/step=5.00 step_ms=51.6 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:57080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:23.771 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=302 last 64 steps: tokens/step=4.67 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:57092 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:27.197 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=353 last 64 steps: tokens/step=4.80 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:57098 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:11:29 [loggers.py:259] Engine 000: Avg prompt throughput: 25.5 tokens/s, Avg generation throughput: 90.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:11:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.80, Accepted throughput: 70.99 tokens/s, Drafted throughput: 93.49 tokens/s, Accepted: 710 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.914, 0.845, 0.770, 0.674, 0.594, Avg Draft acceptance rate: 75.9% |
| (APIServer pid=1) INFO: 172.17.0.1:36472 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:30.699 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=169 last 64 steps: tokens/step=4.28 drafts/step=5.00 step_ms=51.0 host_gap_ms=3.7 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:34.038 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=466 last 64 steps: tokens/step=4.72 drafts/step=5.00 step_ms=51.8 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:36484 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:37.459 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=368 last 64 steps: tokens/step=4.56 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:36494 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:11:39 [loggers.py:259] Engine 000: Avg prompt throughput: 29.2 tokens/s, Avg generation throughput: 82.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:11:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.44, Accepted throughput: 64.00 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 640 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.882, 0.801, 0.683, 0.575, 0.500, Avg Draft acceptance rate: 68.8% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:40.891 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=228 last 64 steps: tokens/step=4.17 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:33870 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:44.300 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=266 last 64 steps: tokens/step=4.27 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:33878 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:33892 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:47.802 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=213 last 64 steps: tokens/step=4.97 drafts/step=5.00 step_ms=51.0 host_gap_ms=3.7 |
| (APIServer pid=1) INFO 10-05 23:11:49 [loggers.py:259] Engine 000: Avg prompt throughput: 25.2 tokens/s, Avg generation throughput: 86.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:11:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.56, Accepted throughput: 66.90 tokens/s, Drafted throughput: 94.00 tokens/s, Accepted: 669 tokens, Drafted: 940 tokens, Per-position acceptance rate: 0.910, 0.793, 0.691, 0.628, 0.537, Avg Draft acceptance rate: 71.2% |
| (APIServer pid=1) INFO: 172.17.0.1:40180 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:51.255 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=153 last 64 steps: tokens/step=5.14 drafts/step=5.00 step_ms=51.9 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:40192 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:54.689 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=122 last 64 steps: tokens/step=4.50 drafts/step=5.00 step_ms=51.7 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:40196 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:11:58.122 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=97 last 64 steps: tokens/step=5.39 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.0 |
| (APIServer pid=1) INFO 10-05 23:11:59 [loggers.py:259] Engine 000: Avg prompt throughput: 30.9 tokens/s, Avg generation throughput: 92.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:11:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.97, Accepted throughput: 73.89 tokens/s, Drafted throughput: 92.99 tokens/s, Accepted: 739 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.925, 0.855, 0.796, 0.731, 0.667, Avg Draft acceptance rate: 79.5% |
| (APIServer pid=1) INFO: 172.17.0.1:58970 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:01.523 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=196 last 64 steps: tokens/step=5.03 drafts/step=5.00 step_ms=51.0 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:58982 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:04.943 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=296 last 64 steps: tokens/step=5.02 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:58998 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:08.364 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=334 last 64 steps: tokens/step=4.84 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:54104 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:12:09 [loggers.py:259] Engine 000: Avg prompt throughput: 37.8 tokens/s, Avg generation throughput: 91.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:12:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.91, Accepted throughput: 72.70 tokens/s, Drafted throughput: 92.99 tokens/s, Accepted: 727 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.925, 0.860, 0.796, 0.699, 0.629, Avg Draft acceptance rate: 78.2% |
| (APIServer pid=1) INFO: 172.17.0.1:54118 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:11.889 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=90 last 64 steps: tokens/step=4.88 drafts/step=5.00 step_ms=51.4 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:54128 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:15.294 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=246 last 64 steps: tokens/step=5.22 drafts/step=5.00 step_ms=51.0 host_gap_ms=2.2 |
| (APIServer pid=1) INFO: 172.17.0.1:54142 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:18.722 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=213 last 64 steps: tokens/step=5.05 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:12:19 [loggers.py:259] Engine 000: Avg prompt throughput: 23.0 tokens/s, Avg generation throughput: 95.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:12:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 5.10, Accepted throughput: 76.60 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 766 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.941, 0.882, 0.834, 0.743, 0.695, Avg Draft acceptance rate: 81.9% |
| (APIServer pid=1) INFO: 172.17.0.1:60328 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:60332 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:22.230 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=113 last 64 steps: tokens/step=4.98 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:60344 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:25.639 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=143 last 64 steps: tokens/step=4.81 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.0 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:28.967 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=401 last 64 steps: tokens/step=3.97 drafts/step=5.00 step_ms=51.6 host_gap_ms=0.4 |
| (APIServer pid=1) INFO 10-05 23:12:29 [loggers.py:259] Engine 000: Avg prompt throughput: 27.1 tokens/s, Avg generation throughput: 85.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:12:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.55, Accepted throughput: 66.30 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 663 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.877, 0.791, 0.717, 0.620, 0.540, Avg Draft acceptance rate: 70.9% |
| (APIServer pid=1) INFO: 172.17.0.1:36526 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:32.387 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=285 last 64 steps: tokens/step=4.39 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:36538 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:35.809 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=348 last 64 steps: tokens/step=4.62 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:36552 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:12:39 [loggers.py:259] Engine 000: Avg prompt throughput: 28.3 tokens/s, Avg generation throughput: 88.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:12:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.76, Accepted throughput: 69.90 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 699 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.930, 0.839, 0.758, 0.651, 0.581, Avg Draft acceptance rate: 75.2% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:39.275 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=182 last 64 steps: tokens/step=5.22 drafts/step=5.00 step_ms=52.1 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:41312 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:42.885 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=343 last 64 steps: tokens/step=4.72 drafts/step=5.00 step_ms=51.5 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:41314 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:41320 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:46.418 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=145 last 64 steps: tokens/step=4.95 drafts/step=5.00 step_ms=51.5 host_gap_ms=3.7 |
| (APIServer pid=1) INFO 10-05 23:12:49 [loggers.py:259] Engine 000: Avg prompt throughput: 38.4 tokens/s, Avg generation throughput: 85.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:12:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.63, Accepted throughput: 66.49 tokens/s, Drafted throughput: 91.49 tokens/s, Accepted: 665 tokens, Drafted: 915 tokens, Per-position acceptance rate: 0.923, 0.809, 0.721, 0.628, 0.552, Avg Draft acceptance rate: 72.7% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:49.749 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=423 last 64 steps: tokens/step=4.34 drafts/step=5.00 step_ms=51.6 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:33408 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:53.176 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=340 last 64 steps: tokens/step=4.58 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:33422 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:12:56.788 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=409 last 64 steps: tokens/step=4.84 drafts/step=5.00 step_ms=51.6 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:33432 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:12:59 [loggers.py:259] Engine 000: Avg prompt throughput: 37.8 tokens/s, Avg generation throughput: 83.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:12:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.67, Accepted throughput: 65.70 tokens/s, Drafted throughput: 89.50 tokens/s, Accepted: 657 tokens, Drafted: 895 tokens, Per-position acceptance rate: 0.894, 0.799, 0.732, 0.665, 0.581, Avg Draft acceptance rate: 73.4% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:00.406 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=382 last 64 steps: tokens/step=4.41 drafts/step=5.00 step_ms=51.7 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:34566 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:03.840 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=254 last 64 steps: tokens/step=4.61 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:34574 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:07.256 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=236 last 64 steps: tokens/step=4.97 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:50106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:13:09 [loggers.py:259] Engine 000: Avg prompt throughput: 24.6 tokens/s, Avg generation throughput: 87.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:13:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.65, Accepted throughput: 68.20 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 682 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.893, 0.813, 0.733, 0.642, 0.567, Avg Draft acceptance rate: 72.9% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:10.672 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=318 last 64 steps: tokens/step=4.98 drafts/step=5.00 step_ms=51.3 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:50122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:50134 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:14.188 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=113 last 64 steps: tokens/step=5.22 drafts/step=5.00 step_ms=51.3 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:50146 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:17.590 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=228 last 64 steps: tokens/step=4.67 drafts/step=5.00 step_ms=51.1 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:13:19 [loggers.py:259] Engine 000: Avg prompt throughput: 26.6 tokens/s, Avg generation throughput: 92.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:13:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.91, Accepted throughput: 73.10 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 731 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.925, 0.845, 0.770, 0.717, 0.652, Avg Draft acceptance rate: 78.2% |
| (APIServer pid=1) INFO: 172.17.0.1:49198 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:21.200 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=301 last 64 steps: tokens/step=5.00 drafts/step=5.00 step_ms=51.5 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:49202 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:24.637 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=263 last 64 steps: tokens/step=4.67 drafts/step=5.00 step_ms=51.7 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:49206 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:56738 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:28.142 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=139 last 64 steps: tokens/step=4.64 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.7 |
| (APIServer pid=1) INFO 10-05 23:13:29 [loggers.py:259] Engine 000: Avg prompt throughput: 42.7 tokens/s, Avg generation throughput: 85.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:13:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.66, Accepted throughput: 66.70 tokens/s, Drafted throughput: 91.00 tokens/s, Accepted: 667 tokens, Drafted: 910 tokens, Per-position acceptance rate: 0.879, 0.813, 0.720, 0.643, 0.610, Avg Draft acceptance rate: 73.3% |
| (APIServer pid=1) INFO: 172.17.0.1:56740 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:31.550 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=301 last 64 steps: tokens/step=5.00 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:56754 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:34.977 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=299 last 64 steps: tokens/step=4.47 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:56760 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:38.422 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=187 last 64 steps: tokens/step=4.88 drafts/step=5.00 step_ms=51.7 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:44080 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:13:39 [loggers.py:259] Engine 000: Avg prompt throughput: 38.6 tokens/s, Avg generation throughput: 88.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:13:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.74, Accepted throughput: 69.10 tokens/s, Drafted throughput: 92.50 tokens/s, Accepted: 691 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.908, 0.805, 0.735, 0.670, 0.616, Avg Draft acceptance rate: 74.7% |
| (APIServer pid=1) INFO: 172.17.0.1:44096 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:41.930 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=162 last 64 steps: tokens/step=5.22 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:44112 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:45.338 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=260 last 64 steps: tokens/step=4.23 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.1 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:48.714 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=537 last 64 steps: tokens/step=4.41 drafts/step=5.00 step_ms=52.3 host_gap_ms=0.4 |
| (APIServer pid=1) INFO 10-05 23:13:49 [loggers.py:259] Engine 000: Avg prompt throughput: 20.5 tokens/s, Avg generation throughput: 84.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:13:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.48, Accepted throughput: 65.50 tokens/s, Drafted throughput: 94.00 tokens/s, Accepted: 655 tokens, Drafted: 940 tokens, Per-position acceptance rate: 0.888, 0.782, 0.686, 0.601, 0.527, Avg Draft acceptance rate: 69.7% |
| (APIServer pid=1) INFO: 172.17.0.1:44122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:52.170 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=296 last 64 steps: tokens/step=4.20 drafts/step=5.00 step_ms=51.9 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:35732 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:55.788 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=310 last 64 steps: tokens/step=4.97 drafts/step=5.00 step_ms=51.7 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:35742 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:48830 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:13:59 [loggers.py:259] Engine 000: Avg prompt throughput: 42.3 tokens/s, Avg generation throughput: 85.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:13:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.72, Accepted throughput: 67.00 tokens/s, Drafted throughput: 90.00 tokens/s, Accepted: 670 tokens, Drafted: 900 tokens, Per-position acceptance rate: 0.889, 0.822, 0.739, 0.683, 0.589, Avg Draft acceptance rate: 74.4% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:13:59.308 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=111 last 64 steps: tokens/step=4.66 drafts/step=5.00 step_ms=51.3 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:48842 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:02.711 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=189 last 64 steps: tokens/step=4.83 drafts/step=5.00 step_ms=51.1 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48846 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:06.130 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=211 last 64 steps: tokens/step=5.23 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48856 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:14:09 [loggers.py:259] Engine 000: Avg prompt throughput: 25.9 tokens/s, Avg generation throughput: 90.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:14:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.80, Accepted throughput: 71.40 tokens/s, Drafted throughput: 94.00 tokens/s, Accepted: 714 tokens, Drafted: 940 tokens, Per-position acceptance rate: 0.910, 0.840, 0.734, 0.681, 0.633, Avg Draft acceptance rate: 76.0% |
| (APIServer pid=1) INFO: 172.17.0.1:48268 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:09.640 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=100 last 64 steps: tokens/step=4.48 drafts/step=5.00 step_ms=51.2 host_gap_ms=3.7 |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:12.963 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=413 last 64 steps: tokens/step=4.89 drafts/step=5.00 step_ms=51.5 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:48274 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:48280 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:16.478 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=180 last 64 steps: tokens/step=4.75 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.8 |
| (APIServer pid=1) INFO: 172.17.0.1:60108 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:14:19 [loggers.py:259] Engine 000: Avg prompt throughput: 34.3 tokens/s, Avg generation throughput: 89.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:14:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.79, Accepted throughput: 70.10 tokens/s, Drafted throughput: 92.50 tokens/s, Accepted: 701 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.930, 0.827, 0.768, 0.692, 0.573, Avg Draft acceptance rate: 75.8% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:19.885 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=296 last 64 steps: tokens/step=4.98 drafts/step=5.00 step_ms=51.2 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:60110 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:23.511 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=465 last 64 steps: tokens/step=4.91 drafts/step=5.00 step_ms=51.8 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:60120 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:26.934 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=342 last 64 steps: tokens/step=4.28 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:14:29 [loggers.py:259] Engine 000: Avg prompt throughput: 26.7 tokens/s, Avg generation throughput: 85.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:14:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.65, Accepted throughput: 67.10 tokens/s, Drafted throughput: 92.00 tokens/s, Accepted: 671 tokens, Drafted: 920 tokens, Per-position acceptance rate: 0.891, 0.804, 0.739, 0.663, 0.549, Avg Draft acceptance rate: 72.9% |
| (APIServer pid=1) INFO: 172.17.0.1:60130 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:30.396 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=223 last 64 steps: tokens/step=4.77 drafts/step=5.00 step_ms=52.0 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:33958 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:33.845 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=141 last 64 steps: tokens/step=5.03 drafts/step=5.00 step_ms=51.8 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:33962 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:37.250 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=169 last 64 steps: tokens/step=4.34 drafts/step=5.00 step_ms=51.1 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:14:39 [loggers.py:259] Engine 000: Avg prompt throughput: 28.9 tokens/s, Avg generation throughput: 89.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:14:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.78, Accepted throughput: 70.70 tokens/s, Drafted throughput: 93.50 tokens/s, Accepted: 707 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.898, 0.834, 0.749, 0.679, 0.620, Avg Draft acceptance rate: 75.6% |
| (APIServer pid=1) INFO: 172.17.0.1:42552 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:40.674 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=189 last 64 steps: tokens/step=5.02 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:42566 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:42580 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:44.169 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=165 last 64 steps: tokens/step=4.91 drafts/step=5.00 step_ms=51.0 host_gap_ms=3.6 |
| (APIServer pid=1) INFO: 172.17.0.1:42594 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:48550 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:47.673 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=87 last 64 steps: tokens/step=5.17 drafts/step=5.00 step_ms=51.1 host_gap_ms=3.7 |
| (APIServer pid=1) INFO 10-05 23:14:49 [loggers.py:259] Engine 000: Avg prompt throughput: 39.7 tokens/s, Avg generation throughput: 91.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:14:49 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.96, Accepted throughput: 72.80 tokens/s, Drafted throughput: 92.00 tokens/s, Accepted: 728 tokens, Drafted: 920 tokens, Per-position acceptance rate: 0.957, 0.880, 0.788, 0.712, 0.620, Avg Draft acceptance rate: 79.1% |
| (APIServer pid=1) INFO: 172.17.0.1:48558 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:51.067 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=219 last 64 steps: tokens/step=5.25 drafts/step=5.00 step_ms=50.9 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48572 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:54.498 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=270 last 64 steps: tokens/step=5.09 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:48584 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:43422 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:14:58.039 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=118 last 64 steps: tokens/step=4.84 drafts/step=5.00 step_ms=51.6 host_gap_ms=3.7 |
| (APIServer pid=1) INFO 10-05 23:14:59 [loggers.py:259] Engine 000: Avg prompt throughput: 43.1 tokens/s, Avg generation throughput: 92.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:14:59 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.97, Accepted throughput: 73.50 tokens/s, Drafted throughput: 92.50 tokens/s, Accepted: 735 tokens, Drafted: 925 tokens, Per-position acceptance rate: 0.930, 0.876, 0.805, 0.708, 0.654, Avg Draft acceptance rate: 79.5% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:01.379 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=425 last 64 steps: tokens/step=4.80 drafts/step=5.00 step_ms=51.6 host_gap_ms=0.5 |
| (APIServer pid=1) INFO: 172.17.0.1:43428 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:04.815 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=306 last 64 steps: tokens/step=4.92 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:43440 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:08.242 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=275 last 64 steps: tokens/step=4.58 drafts/step=5.00 step_ms=51.5 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:33106 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:15:09 [loggers.py:259] Engine 000: Avg prompt throughput: 29.3 tokens/s, Avg generation throughput: 88.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:15:09 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.73, Accepted throughput: 69.40 tokens/s, Drafted throughput: 93.00 tokens/s, Accepted: 694 tokens, Drafted: 930 tokens, Per-position acceptance rate: 0.892, 0.828, 0.737, 0.672, 0.602, Avg Draft acceptance rate: 74.6% |
| (APIServer pid=1) INFO: 172.17.0.1:33122 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:11.764 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=101 last 64 steps: tokens/step=5.22 drafts/step=5.00 step_ms=51.4 host_gap_ms=3.6 |
| (APIServer pid=1) INFO: 172.17.0.1:33138 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:15.163 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=179 last 64 steps: tokens/step=4.72 drafts/step=5.00 step_ms=51.1 host_gap_ms=2.0 |
| (APIServer pid=1) INFO: 172.17.0.1:33148 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:18.583 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=288 last 64 steps: tokens/step=5.08 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:15:19 [loggers.py:259] Engine 000: Avg prompt throughput: 29.0 tokens/s, Avg generation throughput: 90.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:15:19 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.83, Accepted throughput: 71.59 tokens/s, Drafted throughput: 93.49 tokens/s, Accepted: 716 tokens, Drafted: 935 tokens, Per-position acceptance rate: 0.941, 0.840, 0.754, 0.679, 0.615, Avg Draft acceptance rate: 76.6% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:21.956 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=540 last 64 steps: tokens/step=3.95 drafts/step=5.00 step_ms=52.3 host_gap_ms=0.4 |
| (APIServer pid=1) INFO: 172.17.0.1:57536 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:25.529 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=285 last 64 steps: tokens/step=4.53 drafts/step=5.00 step_ms=52.0 host_gap_ms=3.9 |
| (APIServer pid=1) INFO: 172.17.0.1:57546 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:28.952 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=324 last 64 steps: tokens/step=5.31 drafts/step=5.00 step_ms=51.4 host_gap_ms=2.1 |
| (APIServer pid=1) INFO 10-05 23:15:29 [loggers.py:259] Engine 000: Avg prompt throughput: 17.1 tokens/s, Avg generation throughput: 86.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:15:29 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.60, Accepted throughput: 67.69 tokens/s, Drafted throughput: 93.99 tokens/s, Accepted: 677 tokens, Drafted: 940 tokens, Per-position acceptance rate: 0.867, 0.782, 0.718, 0.649, 0.585, Avg Draft acceptance rate: 72.0% |
| (APIServer pid=1) INFO: 172.17.0.1:54288 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:54304 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:32.475 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=117 last 64 steps: tokens/step=4.89 drafts/step=5.00 step_ms=51.3 host_gap_ms=3.7 |
| (APIServer pid=1) INFO: 172.17.0.1:54310 - "POST /v1/completions HTTP/1.1" 200 OK |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:36.064 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=198 last 64 steps: tokens/step=4.42 drafts/step=5.00 step_ms=51.2 host_gap_ms=4.9 |
| (APIServer pid=1) INFO: 172.17.0.1:51808 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:15:39 [loggers.py:259] Engine 000: Avg prompt throughput: 41.5 tokens/s, Avg generation throughput: 86.4 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% |
| (APIServer pid=1) INFO 10-05 23:15:39 [metrics.py:100] SpecDecoding metrics: Mean acceptance length: 4.73, Accepted throughput: 67.79 tokens/s, Drafted throughput: 90.99 tokens/s, Accepted: 678 tokens, Drafted: 910 tokens, Per-position acceptance rate: 0.923, 0.824, 0.736, 0.648, 0.593, Avg Draft acceptance rate: 74.5% |
| (EngineCore_DP0 pid=150) 2026-10-05 23:15:39.500 | INFO | models.demos.gemma4.tt.serving_vllm:speculative_step:331 - speculative pos=209 last 64 steps: tokens/step=5.03 drafts/step=5.00 step_ms=51.6 host_gap_ms=2.1 |
| (APIServer pid=1) INFO: 172.17.0.1:51814 - "POST /v1/completions HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO: 172.17.0.1:51820 - "GET /metrics HTTP/1.1" 200 OK |
| (APIServer pid=1) INFO 10-05 23:15:41 [launcher.py:122] Shutting down FastAPI HTTP server. |
| nanobind: leaked 1566 instances! |
| - leaked instance 0x7b9eb427b8c8 of type "ttnn._ttnn.operations.trace.MeshTraceId" |
| - leaked instance 0x7ba7400f36a8 of type "ttnn._ttnn.tensor.MemoryConfig" |
| - leaked instance 0x7ba7400da4d8 of type "ttnn._ttnn.operations.core.WormholeComputeKernelConfig" |
| - leaked instance 0x7b9ed81aaf78 of type "ttnn._ttnn.operations.normalization.LayerNormShardedMultiCoreProgramConfig" |
| - leaked instance 0x7ba6f41ad338 of type "ttnn._ttnn.tensor.CoreCoord" |
| - leaked instance 0x7ba6f41ac348 of type "ttnn._ttnn.tensor.CoreCoord" |
| - leaked instance 0x7ba7400db9d8 of type "ttnn._ttnn.tensor.Tensor" |
| - leaked instance 0x57d54fcbd300 of type "ttnn._ttnn.operations.unary.lez_t" |
| - leaked instance 0x7ba6f41af8b8 of type "ttnn._ttnn.tensor.CoreCoord" |
| - leaked instance 0x7b9ee0179948 of type "ttnn._ttnn.operations.normalization.LayerNormShardedMultiCoreProgramConfig" |
| - leaked instance 0x7ba7400f12c8 of type "ttnn._ttnn.tensor.MemoryConfig" |
| - leaked instance 0x7ba740114c28 of type "ttnn._ttnn.operations.normalization.LayerNormShardedMultiCoreProgramConfig" |
| - leaked instance 0x7ba7400f3108 of type "ttnn._ttnn.tensor.MemoryConfig" |
| - leaked instance 0x57d5502d6ca0 of type "ttnn._ttnn.operations.unary.eqz_t" |
| - leaked instance 0x7ba7478f4078 of type "ttnn._ttnn.tensor.Tensor" |
| - leaked instance 0x7ba74786b558 of type "ttnn._ttnn.tensor.Tensor" |
| - leaked instance 0x7ba7400793f8 of type "ttnn._ttnn.tensor.Tensor" |
| - leaked instance 0x7ba6f41ae7d8 of type "ttnn._ttnn.tensor.CoreCoord" |
| - leaked instance 0x7ba74786aef8 of type "ttnn._ttnn.tensor.Tensor" |
| - leaked instance 0x7ba7400f10a8 of type "ttnn._ttnn.tensor.MemoryConfig" |
| - ... skipped remainder |
| nanobind: leaked 130 keep_alive records! |
| nanobind: leaked 945 types! |
| - leaked type "SocketEndpoint" |
| - leaked type "ttnn._ttnn.operations.experimental.quasar.gt__t" |
| - leaked type "ttnn._ttnn.d2d_stream_service.D2DStreamServiceSender" |
| - leaked type "ttnn._ttnn.operations.experimental.quasar.SliceDeviceOperation" |
| - leaked type "ttnn._ttnn.operations.experimental.quasar.MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig" |
| - leaked type "ttnn._ttnn.operations.binary.lt__t" |
| - leaked type "ttnn._ttnn.operations.binary_backward.xlogy_bw_t" |
| - leaked type "ttnn._ttnn.operations.experimental.insert_t" |
| - leaked type "ttnn._ttnn.operations.experimental.quasar.isclose_t" |
| - leaked type "ttnn._ttnn.operations.reduction.var_t" |
| - ... skipped remainder |
| nanobind: leaked 4351 functions! |
| - leaked function "" |
| - leaked function "" |
| - leaked function "" |
| - leaked function "" |
| - leaked function "sender_cores" |
| - leaked function "__init__" |
| - leaked function "__call__" |
| - leaked function "track_function_start" |
| - leaked function "" |
| - leaked function "" |
| - ... skipped remainder |
| nanobind: this is likely caused by a reference counting issue in the binding code. |
| See https: |
| 2026-10-05 23:15:41.529 | info | BuildKernels | JIT telemetry: 3 registered TelemetryTokens (build_cache_telemetry.cpp:230) |
| 2026-10-05 23:15:41.553 | info | Device | Closing user mode device drivers (tt_cluster.cpp:513) |
| 2026-10-05 23:15:41.553 | info | UMD | Closing devices in cluster (cluster.cpp:1105) |
| 2026-10-05 23:15:41.650 | info | UMD | Closing devices in cluster completed. (cluster.cpp:1114) |
| 2026-10-05 23:15:41.650 | info | UMD | Cluster destructor started. (cluster.cpp:808) |
| 2026-10-05 23:15:41.650 | info | UMD | Cluster destructor completed. (cluster.cpp:811) |
| (APIServer pid=1) INFO: Shutting down |
| (APIServer pid=1) INFO: Waiting for application shutdown. |
| (APIServer pid=1) INFO: Application shutdown complete. |
| nanobind: leaked 99 instances! |
| - leaked instance 0x74f45c59c0a8 of type "ttnn._ttnn.tensor.MemoryConfig" |
| - leaked instance 0x613bffd8a390 of type "ttnn._ttnn.operations.unary.isposinf_t" |
| - leaked instance 0x613c01cda150 of type "ttnn._ttnn.operations.unary.asinh_t" |
| - leaked instance 0x613c01695ca0 of type "ttnn._ttnn.operations.unary.exp_t" |
| - leaked instance 0x74f45c420738 of type "ttnn._ttnn.operations.experimental.IndexerScoreProgramConfig" |
| - leaked instance 0x74f45c571998 of type "ttnn._ttnn.tensor.CoreRangeSet" |
| - leaked instance 0x613bffd87e60 of type "ttnn._ttnn.operations.unary.isinf_t" |
| - leaked instance 0x74f45c44dac8 of type "ttnn._ttnn.operations.sliding_window.Op2DSliceConfig" |
| - leaked instance 0x613c01861150 of type "ttnn._ttnn.operations.unary.mish_t" |
| - leaked instance 0x74f45c618f88 of type "ttnn._ttnn.tensor.MemoryConfig" |
| - leaked instance 0x74f45c59c048 of type "ttnn._ttnn.tensor.MemoryConfig" |
| - leaked instance 0x613bffd8f820 of type "ttnn._ttnn.operations.unary.silu_t" |
| - leaked instance 0x74f45c61a6e8 of type "ttnn._ttnn.operations.normalization.SoftmaxDefaultProgramConfig" |
| - leaked instance 0x613c016f0a80 of type "ttnn._ttnn.operations.unary.erfc_t" |
| - leaked instance 0x74f45c4200a8 of type "ttnn._ttnn.tensor.CoreRangeSet" |
| - leaked instance 0x613c01448fc0 of type "ttnn._ttnn.operations.unary.ltz_t" |
| - leaked instance 0x613c01c92ea0 of type "ttnn._ttnn.operations.unary.floor_t" |
| - leaked instance 0x613c01694780 of type "ttnn._ttnn.operations.unary.tanh_t" |
| - leaked instance 0x74f45c618f08 of type "ttnn._ttnn.tensor.MemoryConfig" |
| - leaked instance 0x74f45c61ac48 of type "ttnn._ttnn.operations.normalization.SoftmaxDefaultProgramConfig" |
| - ... skipped remainder |
| nanobind: leaked 942 types! |
| - leaked type "SocketEndpoint" |
| - leaked type "ttnn._ttnn.operations.experimental.quasar.gt__t" |
| - leaked type "ttnn._ttnn.d2d_stream_service.D2DStreamServiceSender" |
| - leaked type "ttnn._ttnn.operations.experimental.quasar.SliceDeviceOperation" |
| - leaked type "ttnn._ttnn.operations.experimental.quasar.MatmulMultiCoreReuseMultiCastBatchedDRAMShardedProgramConfig" |
| - leaked type "ttnn._ttnn.operations.binary.lt__t" |
| - leaked type "ttnn._ttnn.operations.binary_backward.xlogy_bw_t" |
| - leaked type "ttnn._ttnn.operations.experimental.insert_t" |
| - leaked type "ttnn._ttnn.operations.experimental.quasar.isclose_t" |
| - leaked type "ttnn._ttnn.operations.reduction.var_t" |
| - ... skipped remainder |
| nanobind: leaked 4345 functions! |
| - leaked function "" |
| - leaked function "" |
| - leaked function "get_active_cores" |
| - leaked function "" |
| - leaked function "__call__" |
| - leaked function "" |
| - leaked function "" |
| - leaked function "" |
| - leaked function "__ge__" |
| - leaked function "" |
| - ... skipped remainder |
| nanobind: this is likely caused by a reference counting issue in the binding code. |
| See https: |
