Instructions to use vcruz305/DeepSeek-V4.1-Flash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use vcruz305/DeepSeek-V4.1-Flash-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
Use Docker
docker model run hf.co/vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use vcruz305/DeepSeek-V4.1-Flash-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "vcruz305/DeepSeek-V4.1-Flash-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vcruz305/DeepSeek-V4.1-Flash-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
- Ollama
How to use vcruz305/DeepSeek-V4.1-Flash-GGUF with Ollama:
ollama run hf.co/vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use vcruz305/DeepSeek-V4.1-Flash-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use vcruz305/DeepSeek-V4.1-Flash-GGUF with Docker Model Runner:
docker model run hf.co/vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
- Lemonade
How to use vcruz305/DeepSeek-V4.1-Flash-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.DeepSeek-V4.1-Flash-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use vcruz305/DeepSeek-V4.1-Flash-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use vcruz305/DeepSeek-V4.1-Flash-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "vcruz305/DeepSeek-V4.1-Flash-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Upload llama.cpp/patches/patch_llamacpp_v41.py with huggingface_hub
Browse files
llama.cpp/patches/patch_llamacpp_v41.py
CHANGED
|
@@ -79,6 +79,19 @@ CONST_TENSOR_NAME_NEW = """ MODEL_TENSOR.ENGRAM_EMBD: "blk.{bid
|
|
| 79 |
# The list below is derived from the 39 tensor families the converter actually wrote,
|
| 80 |
# reverse mapped through TENSOR_NAMES, rather than copied from DEEPSEEK4 and trimmed.
|
| 81 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
CONST_ARCH_ENUM_ANCHOR = " DEEPSEEK4 = auto()\n"
|
| 83 |
CONST_ARCH_ENUM_NEW = """ DEEPSEEK4 = auto()
|
| 84 |
DEEPSEEK41 = auto()
|
|
@@ -231,10 +244,14 @@ class DeepseekV41Model(DeepseekV4Model):
|
|
| 231 |
super().set_gguf_parameters()
|
| 232 |
hparams = self.hparams
|
| 233 |
if (engram_ids := hparams.get("engram_layer_ids")) is not None:
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
self.gguf_writer.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 238 |
|
| 239 |
|
| 240 |
# rows per block when rewriting an engram table; 1M rows is about 1 GB of float32 scratch
|
|
@@ -378,6 +395,7 @@ def patch_constants(path: pathlib.Path, revert: bool, check: bool) -> int:
|
|
| 378 |
anchors = (
|
| 379 |
CONST_TENSOR_ENUM_ANCHOR,
|
| 380 |
CONST_TENSOR_NAME_ANCHOR,
|
|
|
|
| 381 |
CONST_ARCH_ENUM_ANCHOR,
|
| 382 |
CONST_ARCH_NAME_ANCHOR,
|
| 383 |
CONST_ARCH_TENSORS_ANCHOR,
|
|
@@ -395,6 +413,8 @@ def patch_constants(path: pathlib.Path, revert: bool, check: bool) -> int:
|
|
| 395 |
if "ENGRAM_EMBD" not in text:
|
| 396 |
text = text.replace(CONST_TENSOR_ENUM_ANCHOR, CONST_TENSOR_ENUM_NEW + CONST_TENSOR_ENUM_ANCHOR, 1)
|
| 397 |
text = text.replace(CONST_TENSOR_NAME_ANCHOR, CONST_TENSOR_NAME_NEW + CONST_TENSOR_NAME_ANCHOR, 1)
|
|
|
|
|
|
|
| 398 |
text = text.replace(CONST_ARCH_ENUM_ANCHOR, CONST_ARCH_ENUM_NEW, 1)
|
| 399 |
text = text.replace(CONST_ARCH_NAME_ANCHOR, CONST_ARCH_NAME_NEW, 1)
|
| 400 |
text = text.replace(CONST_ARCH_TENSORS_ANCHOR, CONST_ARCH_TENSORS_NEW, 1)
|
|
|
|
| 79 |
# The list below is derived from the 39 tensor families the converter actually wrote,
|
| 80 |
# reverse mapped through TENSOR_NAMES, rather than copied from DEEPSEEK4 and trimmed.
|
| 81 |
|
| 82 |
+
# The four engram KV keys, declared the way every other arch-scoped key is: with an {arch}
|
| 83 |
+
# placeholder. PerLayerEmbedding directly below is the closest existing analogue, and llama.cpp's
|
| 84 |
+
# own lazy-read comment pairs PLE and engrams for the same reason.
|
| 85 |
+
CONST_KV_ANCHOR = " class PerLayerEmbedding:\n"
|
| 86 |
+
CONST_KV_NEW = ''' class Engram:
|
| 87 |
+
LAYER_IDS = "{arch}.engram.layer_ids"
|
| 88 |
+
HEAD_COUNT = "{arch}.engram.head_count"
|
| 89 |
+
KEY_LENGTH = "{arch}.engram.key_length"
|
| 90 |
+
MAX_NGRAM_SIZE = "{arch}.engram.max_ngram_size"
|
| 91 |
+
|
| 92 |
+
class PerLayerEmbedding:
|
| 93 |
+
'''
|
| 94 |
+
|
| 95 |
CONST_ARCH_ENUM_ANCHOR = " DEEPSEEK4 = auto()\n"
|
| 96 |
CONST_ARCH_ENUM_NEW = """ DEEPSEEK4 = auto()
|
| 97 |
DEEPSEEK41 = auto()
|
|
|
|
| 244 |
super().set_gguf_parameters()
|
| 245 |
hparams = self.hparams
|
| 246 |
if (engram_ids := hparams.get("engram_layer_ids")) is not None:
|
| 247 |
+
# These MUST carry the arch prefix, not a literal "deepseek4.". llama.cpp resolves
|
| 248 |
+
# every LLM_KV_* as "{arch}.{key}", so a hardcoded prefix means the runtime looks up
|
| 249 |
+
# deepseek41.engram.head_count and finds nothing, on a file that otherwise loads.
|
| 250 |
+
arch = self.gguf_writer.arch
|
| 251 |
+
self.gguf_writer.add_uint32(gguf.Keys.Engram.HEAD_COUNT.format(arch=arch), hparams["engram_n_heads"])
|
| 252 |
+
self.gguf_writer.add_uint32(gguf.Keys.Engram.KEY_LENGTH.format(arch=arch), hparams["engram_head_dim"])
|
| 253 |
+
self.gguf_writer.add_uint32(gguf.Keys.Engram.MAX_NGRAM_SIZE.format(arch=arch), hparams["engram_max_ngram_size"])
|
| 254 |
+
self.gguf_writer.add_array(gguf.Keys.Engram.LAYER_IDS.format(arch=arch), engram_ids)
|
| 255 |
|
| 256 |
|
| 257 |
# rows per block when rewriting an engram table; 1M rows is about 1 GB of float32 scratch
|
|
|
|
| 395 |
anchors = (
|
| 396 |
CONST_TENSOR_ENUM_ANCHOR,
|
| 397 |
CONST_TENSOR_NAME_ANCHOR,
|
| 398 |
+
CONST_KV_ANCHOR,
|
| 399 |
CONST_ARCH_ENUM_ANCHOR,
|
| 400 |
CONST_ARCH_NAME_ANCHOR,
|
| 401 |
CONST_ARCH_TENSORS_ANCHOR,
|
|
|
|
| 413 |
if "ENGRAM_EMBD" not in text:
|
| 414 |
text = text.replace(CONST_TENSOR_ENUM_ANCHOR, CONST_TENSOR_ENUM_NEW + CONST_TENSOR_ENUM_ANCHOR, 1)
|
| 415 |
text = text.replace(CONST_TENSOR_NAME_ANCHOR, CONST_TENSOR_NAME_NEW + CONST_TENSOR_NAME_ANCHOR, 1)
|
| 416 |
+
if "class Engram:" not in text:
|
| 417 |
+
text = text.replace(CONST_KV_ANCHOR, CONST_KV_NEW, 1)
|
| 418 |
text = text.replace(CONST_ARCH_ENUM_ANCHOR, CONST_ARCH_ENUM_NEW, 1)
|
| 419 |
text = text.replace(CONST_ARCH_NAME_ANCHOR, CONST_ARCH_NAME_NEW, 1)
|
| 420 |
text = text.replace(CONST_ARCH_TENSORS_ANCHOR, CONST_ARCH_TENSORS_NEW, 1)
|