local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE

DeepSeek V4.1 Flash 1017

Run this prepared Flare runtime on four NVIDIA SM120 GPUs with 96 GB VRAM each. It supports 32 concurrent requests, five DSpark draft tokens per round, text and images, and a 1,048,576-token per-request context limit. Serving uses Flare's native Rust runtime without Python or runtime compilation.

The package contains model weights, compiled CUDA kernels and captured execution graphs. It requires the Flare image below; Transformers, vLLM, SGLang and llama.cpp cannot load this runtime format. MXFP4 expert weights retain losslessly compressed CSF scales, attention uses BF16 arithmetic, and KV storage uses FP8 compression.

Use Linux x86-64, the NVIDIA Container Toolkit and working CUDA peer access between all four GPUs. Four RTX PRO 6000 Blackwell Workstation Edition GPUs were used for serving checks. Prefer GPUs behind the same PCIe switch. Allow about 515 GB of disk space for all nine runtime files, plus download-cache headroom. The examples keep the 188.8 GiB Engram tables in system RAM; allow at least 256 GiB of host RAM for the tables and serving overhead.

The public main revision contains the model card, license and publisher profile. Runtime files are in gb202. Keep the signed primary file and all eight data companions together with their original filenames.

Download and run

Install the Hugging Face hf CLI and download the complete C32 runtime:

mkdir -p models flare-state
hf download local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \
  --revision gb202 \
  --include 'ds41-1017-tp4-c32-d5-packed64-bf16-default-sampling*.safetensors' \
  --local-dir ./models

Run the primary file in Docker:

docker run --rm -it --user "$(id -u):$(id -g)" \
  --gpus '"device=0,1,2,3"' --ulimit memlock=-1:-1 \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$PWD/models,dst=/models,readonly" \
  --mount "type=bind,src=$PWD/flare-state,dst=/state" \
  ghcr.io/local-inference-lab/flare:83d582e2-amd64 \
  run /models/ds41-1017-tp4-c32-d5-packed64-bf16-default-sampling.safetensors \
  --gpus 0,1,2,3 --max-concurrency 32 --mtp-draft-tokens 5 \
  --mtp-adaptive-verification --scheduler-policy throughput \
  --embedding-table-storage ram --kv-cache-limit memory \
  --trust-store /state/trust.toml --host 0.0.0.0 --port 8000

Review and accept the publisher and model license when prompted. The writable state directory preserves that acceptance. Docker selects host GPUs 0–3; Flare addresses them by their container-local indices.

With a compatible Flare executable installed directly on the host:

flare run ./models/ds41-1017-tp4-c32-d5-packed64-bf16-default-sampling.safetensors \
  --gpus 0,1,2,3 --max-concurrency 32 --mtp-draft-tokens 5 \
  --mtp-adaptive-verification --scheduler-policy throughput \
  --embedding-table-storage ram --kv-cache-limit memory \
  --host 127.0.0.1 --port 8000

Download through Flare

Flare can select gb202, download the complete package and serve it from a persistent cache. The explicit primary filename selects this C32 runtime:

mkdir -p flare-cache flare-state
docker run --rm -it --user "$(id -u):$(id -g)" \
  --gpus '"device=0,1,2,3"' --ulimit memlock=-1:-1 \
  -p 127.0.0.1:8000:8000 \
  --mount "type=bind,src=$PWD/flare-cache,dst=/cache" \
  --mount "type=bind,src=$PWD/flare-state,dst=/state" \
  ghcr.io/local-inference-lab/flare:83d582e2-amd64 \
  run local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE \
  --filename ds41-1017-tp4-c32-d5-packed64-bf16-default-sampling.safetensors \
  --gpus 0,1,2,3 --max-concurrency 32 --mtp-draft-tokens 5 \
  --mtp-adaptive-verification --scheduler-policy throughput \
  --embedding-table-storage ram --kv-cache-limit memory --cache-dir /cache \
  --trust-store /state/trust.toml --host 0.0.0.0 --port 8000

For download only, replace run with download and omit the serving flags. This repository is public. If authentication is required, run hf auth login, mount the token file read-only and pass its container path with --token-file.

Send a request

The OpenAI-compatible API serves model ID deepseek-v4.1-flash:

curl --fail http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"Explain how a hash table works."}],"max_tokens":1024}'

Sampling defaults are temperature 1.0, top-p 1.0, top-k disabled, min-p 0, repetition penalty 1.0, and zero presence/frequency penalties. Request values override them. For nucleus sampling, add "top_p":0.95; top-k and min-p can be set with "top_k":17 and "min_p":0.05.

Images use OpenAI image_url content items with PNG, JPEG or static WebP data URLs. The package supports four simultaneous image-bearing requests within its 32-request total. Each request shares a 4096-visual-token budget across its images; usable image count depends on resolution. Image requests queue when all four feature slots are occupied.

Serving settings

• --max-concurrency defaults to 32. Keep 32 for aggregate throughput; use --max-concurrency 8 to reduce resident execution state. The package cannot exceed 32 active requests.
• --mtp-draft-tokens defaults to 5, with adaptive verification enabled. Keep these settings for the packed graph catalog. Use --mtp-draft-tokens 0 for target-only execution. Positive draft limits 1–4 and fixed-width verification are not supported by this package.
• --scheduler-policy defaults to balanced. The examples recommend throughput for concurrent generation. Use --scheduler-policy balanced to select the default latency/throughput balance.
• --kv-cache-limit defaults to memory, as shown above. Available GPU memory determines the KV pool. On the measured four-GPU host, the runtime allocated 21,160,832 logical KV tokens shared across requests. This capacity is not multiplied by the TP rank count. The per-request context limit remains 1,048,576 tokens.
• --memory-limit defaults to the package's zero serving reserve, apart from Flare's driver margin. Dedicated GPUs provide the largest KV pool. To retain 2 GiB per GPU for other work, add --memory-limit=-2G; the KV pool will shrink accordingly.
• Prefill uses captured chunks through 4096 tokens. Changing this prepared limit requires a different runtime package.
• --embedding-table-storage defaults to disk. The examples recommend ram to avoid Engram disk reads during generation. RAM mode keeps one host copy of the tables and transfers the required rows to each rank.

Disk-backed Engram

To avoid holding the full Engram tables in RAM, use fast local NVMe storage with a bounded row cache:

flare run ./models/ds41-1017-tp4-c32-d5-packed64-bf16-default-sampling.safetensors \
  --gpus 0,1,2,3 --max-concurrency 32 --mtp-draft-tokens 5 \
  --mtp-adaptive-verification --scheduler-policy throughput \
  --embedding-table-storage disk --embedding-table-cache-size 8GiB \
  --kv-cache-limit memory --host 127.0.0.1 --port 8000

--embedding-table-cache-size defaults to zero and applies only to disk storage; it cannot be combined with RAM storage. Disk-backed performance depends on storage latency. Docker containers using disk-backed tables must allow Linux io_uring operations. Engram offload leaves transformer weights in GPU memory.

Native serving measurements

Concurrent requests	Aggregate decode tokens/s	Measured cells
8	891.2	4
32	1473.5	2

Measured on four RTX PRO 6000 Blackwell Workstation Edition GPUs with 96 GB each, BF16 MLA, CSF expert scales, FP8 KV, RAM Engram, adaptive packed verification through 64 rows and the throughput scheduler. Requests used 91-token prompts, temperature 1.0/top-p 1.0, disabled top-k and min-p 0; each cell measured 30 seconds using streamed usage. Mean active SM clocks were 2800.6 MHz at concurrency eight and 2795.1 MHz at concurrency 32; memory clocks were 13365 MHz. Measurements used the native CLI and this signed package; Docker serving was verified separately with ghcr.io/local-inference-lab/flare:83d582e2-amd64.

Text, image order, prefix reuse, mixed request policies, chunked prefill and JSON output have serving smoke coverage. Comprehensive model-accuracy evaluation has not been run.