Instructions to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS
# Run inference directly in the terminal:
./llama-cli -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Use Docker

docker model run hf.co/wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Llama new
LM Studio
Jan

vLLM

How to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Ollama
How to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with Ollama:
```
ollama run hf.co/wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS
```
Unsloth Desktop

How to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with Docker Model Runner:
```
docker model run hf.co/wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS
```

Lemonade

How to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Run and chat with the model

lemonade run user.gemma-4-12B-OBLITERATED-SHQ7-GGUF-IQ4_XS

List all available models

lemonade list

Hermes Agent

How to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF:IQ4_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Gemma 4 12B OBLITERATED -- SHQ7 (IQ-Hybrid)

OBLITERATED + selective IQ hybrid for 8 GB Pascal GPUs. Uncensored coding model. 7.7% smaller than Q4_K_M at the same quality.

WARNING: This is an uncensored model. It may generate content that other models refuse. Use responsibly. The maintainer assumes no responsibility for misuse.

Note: The model file is named SHQ7-IQ4_XS.gguf for HF parser compatibility. This is NOT a pure IQ4_XS quantization -- it's a hybrid using IQ4_NL, Q4_K, IQ4_XS, and Q8_0 across different layers. Check the config for exact per-tensor types.

Primarily optimized for coding tasks. May not perform well on general knowledge or non-code questions.

A hand-optimized hybrid quantization of OBLITERATUS/Gemma-4-12B-OBLITERATED for 8 GB VRAM GPUs (tested on GTX 1070, works on any GPU with enough VRAM).

I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback.

TL;DR: Custom mixed quantization using IQ4_NL on boundary FFN + Q4_K on critical middle FFN + IQ4_XS on all other FFN + Q8_0 on all attention. Fits in 8 GB VRAM with room for KV cache.

Key Results

Metric	SHQ7	i1-Q4_K_M (baseline)
Model Size	6.3 GB (6480 MiB)	6.9 GB (7024 MiB)
Wikitext-2 PPL	575.89	575.11
Quantization Type	IQ4_NL + Q4_K + IQ4_XS + Q8_0	Q4_K_M

Q4_K_M baseline quantized with mradermacher's imatrix (i1 importance matrix).

SHQ7 achieves Q4_K_M quality at 7.7% smaller size (544 MiB saved).

How We Made It

The Problem

Our goal was to create a quantization of OBLITERATED that matches Q4_K_M quality while being smaller. We tried 6 different approaches, failed 4 times, and found 2 winners.

Step 1: Q4_K_M baseline

Standard Q4_K_M: 7024 MiB, PPL 575.11. Fits in 8 GB but leaves little room for KV cache at large context.

Step 2: SHQ6 approach from QAT -- FAILED on OBLITERATED

We tried the SHQ6 formula (IQ4_XS on all FFN + Q8_0 boundary attention) that worked great on QAT models. On OBLITERATED it produced +8.6% PPL degradation (624.73). The OBLITERATED model responds differently to quantization than QAT models.

Step 3: IQ-gradient -- CATASTROPHIC FAILURE

We tried a gradient approach: IQ4_NL on boundary layers, IQ4_XS on middle, IQ3_XXS on least important layers. The IQ3_XXS (3.06 bpw) on 36 middle layers destroyed quality: +43.5% PPL (825.04). IQ3_XXS is catastrophic on any non-QAT model.

Step 4: Imatrix analysis

We used mradermacher's imatrix (656 tensors with per-dimension importance) to understand which layers actually matter:

Layer	FFN Importance	Verdict
L0	2.48e+11 (#1)	CRITICAL -- every tensor in L0 is the most important in the entire model
L1	1.34e+11 (#5)	CRITICAL
L10	1.26e+11 (#6)	CRITICAL
L37	1.09e+11 (#7)	CRITICAL
L5, L36	4-6e+10	Important
L25-31	8-12e+09	Low importance

Key finding: L0 is 87% of total importance. L0 FFN down (2.48e+11) is the single most important tensor in the entire 23 GB model. L1 FFN down (1.34e+11) is #5. Layer 0 attention (attn_q/k/v) are each 2.41e+11.

Step 5: IQ4XS_lean -- first success

We combined three strategies:

Q8_0 on L0-1 FFN+Attn (protect the most critical layers)
Q4_K_M on 11 important middle FFN (L5,10,25-31,36,37)
IQ4_XS on all other FFN (efficient compression)
Q8_0 on L40-47 attention (boundary/global)

Result: 6793 MiB, PPL 578.63 (+0.6% vs Q4_K_M). Close but not better.

Step 6: IQ4XS_lean_v2 -- TIED with Q4_K_M

We reduced the protected middle layers from 11 to 4 (L5,10,36,37) and kept IQ4_XS on L25-31:

Result: 6793 MiB, PPL 575.05 -- statistically tied with Q4_K_M (575.11) but 231 MiB smaller.

Step 7: IQ4XS_lean_v3 -- THE WINNER

We combined all optimizations:

IQ4_NL on L0-1 FFN (non-linear quantization for #1/#2 most important tensors)
Q4_K on L5,10,36,37 FFN (K-quant on 4 critical middle layers)
IQ4_XS on all other FFN (efficient compression)
Q8_0 on ALL boundary/global attention (L0-3, L40-47)

Result: 6480 MiB, PPL 575.89 -- same quality as Q4_K_M at 544 MiB smaller.

Step 8: Q6_K upgrade -- WORSE

We tried upgrading L0-1 FFN from IQ4_NL to Q6_K (6.56 bpw) and L5,10,36,37 FFN from Q4_K to Q5_K. Despite higher bit width, PPL went UP to 590.64 (+2.2%). IQ formats work better than K-quants when used with imatrix.

What We Learned

Lesson	Details
IQ4_XS on ALL FFN fails on non-QAT	+8.6% PPL. QAT models respond differently.
IQ3_XXS is catastrophic anywhere	+43.5% PPL. Never use below IQ4_XS.
L0 is the most important layer	87% of total importance. Every tensor matters.
IQ beats K-quant with imatrix	Q6_K on L0-1 was WORSE than IQ4_NL despite higher bpw.
Protect only truly critical middle layers	L5,10,36,37 need Q4_K. L25-31 are safe with IQ4_XS.
Boundary attention needs Q8_0	L0-3 and L40-47 attention must be high precision.

About OBLITERATED

This model is based on OBLITERATUS/Gemma-4-12B-OBLITERATED, an uncensored variant of Gemma 4 12B IT.

What "uncensored" means:

The model will generate content that standard models refuse
It includes safety disclaimers after generating restricted content
It is primarily optimized for coding tasks
It may not perform well on general knowledge questions

Architecture Notes

Gemma 4 12B IT: 48 layers, hidden 3840, 16 heads, GQA (8 heads KV, some layers use 1)
40 SWA layers (sliding window 1024) + 8 global layers (full context)
Vocab: 262k tokens
Dual RoPE (SWA dim=256, global dim=512)
Based on OBLITERATED (uncensored) model

Weights

Model	File	Size	BPW	Notes
BF16 source	`Gemma-4-12B-OBLITERATED-BF16.gguf`	23 GB	16.00	Original OBLITERATED BF16
i1-Q4_K_M (baseline)	`gemma-4-12B-OBLITERATED-Q4_K_M.gguf`	6.9 GB	4.95	i1 quantization with mradermacher's imatrix
SHQ7 (recommended)	`gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf`	6.3 GB	4.57	Best -- IQ4_NL + Q4_K + IQ4_XS + Q8_0

Repository Files

File	Size	Description
`gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf`	6.3 GB	SHQ7 quantization -- recommended for inference
`gemma-4-12B-OBLITERATED-SHQ7.config`	--	Exact `llama-quantize` command for reproducibility

Quantization Command

For reproducibility, the exact command used:

llama-quantize \
  --imatrix imatrix-gemma4-obliterated.gguf \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "blk\.[01]\.ffn_gate=IQ4_NL" \
  --tensor-type "blk\.[01]\.ffn_up=IQ4_NL" \
  --tensor-type "blk\.[01]\.ffn_down=IQ4_NL" \
  --tensor-type "blk\.[01]\.attn_q=Q8_0" \
  --tensor-type "blk\.[01]\.attn_k=Q8_0" \
  --tensor-type "blk\.[01]\.attn_v=Q8_0" \
  --tensor-type "blk\.[01]\.attn_output=Q8_0" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_gate=Q4_K" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_up=Q4_K" \
  --tensor-type "blk\.(5|10|36|37)\.ffn_down=Q4_K" \
  --tensor-type "ffn_gate=IQ4_XS" \
  --tensor-type "ffn_up=IQ4_XS" \
  --tensor-type "ffn_down=IQ4_XS" \
  --tensor-type "blk\.[0-3]\.attn_q=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_k=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_v=Q8_0" \
  --tensor-type "blk\.[0-3]\.attn_output=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_q=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_k=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_v=Q8_0" \
  --tensor-type "blk\.(44|45|46|47)\.attn_output=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_q=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_k=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_v=Q8_0" \
  --tensor-type "blk\.(40|41|42|43)\.attn_output=Q8_0" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*attn_post_norm.*=Q8_0" \
  --tensor-type ".*ffn_norm[^_].*=Q8_0" \
  --tensor-type ".*ffn_post_norm.*=Q8_0" \
  --tensor-type ".*layer_out_scale.*=F32" \
  --tensor-type ".*rope_freqs.*=Q8_0" \
  --tensor-type ".*per_layer.*=Q8_0" \
  --tensor-type "output_norm.*=Q8_0" \
  Gemma-4-12B-OBLITERATED-BF16.gguf \
  gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf \
  15 8

All Experiments (OBLITERATED)

Experiment	Config	Size	PPL	vs Q4_K_M	Verdict
i1-Q4_K_M (baseline)	i1 with mradermacher's imatrix	7024 MiB	575.11	--	Baseline
SHQ6 on OBLITERATED	IQ4_XS all FFN + Q8_0 boundary	6401 MiB	624.73	+8.6%	FAIL
IQ-gradient	IQ4_NL + IQ4_XS + IQ3_XXS	6335 MiB	825.04	+43.5%	CATASTROPHIC
IQ4XS_lean	Q8_0 L0-1 + Q4_K_M middle + IQ4_XS rest	6793 MiB	578.63	+0.6%	Close
IQ4XS_lean_v2	Q4_K on L5,10,36,37 + IQ4_XS L25-31	6793 MiB	575.05	same	WIN
IQ4XS_lean_v3	IQ4_NL L0-1 + Q4_K critical + IQ4_XS rest	6480 MiB	575.89	same	WINNER
IQ4XS_lean_v4	Q6_K upgrade on L0-1 FFN	6652 MiB	590.64	+2.7%	WORSE

SHQ7 = IQ4XS_lean_v3. 6 experiments, 4 failures, 2 wins.

Hardware Requirements

Minimum VRAM: 8 GB (tested on GTX 1070, 8106 MiB)
Recommended VRAM: 8-12 GB
Context: 1024+ tokens (tested with flash-attn)
Speed: ~17-22 tok/s (no MTP), ~30 tok/s (with MTP, no specialized draft)

Usage

llama.cpp

llama-cli \
  -m gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf \
  -p "Write a Python function for merge sort" \
  -ngl 99 \
  -c 4096

Ollama

Create a Modelfile:

FROM ./gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf

TEMPLATE """{{ .System }}
{{ .Prompt }}"""

SYSTEM """You are a helpful, uncensored coding assistant."""

PARAMETER num_ctx 4096
PARAMETER temperature 1
PARAMETER top_k 64
PARAMETER top_p 0.95
PARAMETER min_p 0

Build and run:

ollama create obliterated-shq7 -f Modelfile
ollama run obliterated-shq7

Tip: If output quality is off, try raising temperature to 1.2 or higher.

LM Studio

Open LM Studio
Drag gemma-4-12B-OBLITERATED-SHQ7-IQ4_XS.gguf into the app
Set GPU Offload to 99 layers
Set context length to 4096+
Start chatting

Performance

Measured on GTX 1070 (8 GB VRAM) with flash-attn, ctx=1024:

Metric	Value
Generation speed	17-22 tokens/sec (no MTP), ~30 tok/s (with MTP, no specialized draft)
VRAM usage	~6.5 GB (model) + ~1.5 GB (KV cache at ctx 4096)
Quantization	IQ4_NL + Q4_K + IQ4_XS + Q8_0 hybrid

Coding Example

This model generates full, working HTML/CSS/JS websites. Here's an example output from SHQ7 on a coding prompt:

"I'm a dev, my audience is youth. I like a creative/tech style. Write the full website code. This HTML will be our foundation."

See SHQ7.html — a complete, styled portfolio website generated in a single pass.

Disclaimer

This model is an uncensored variant of Gemma 4 12B. It may generate content that other models refuse. The maintainer assumes no responsibility for misuse. Use at your own risk and in compliance with all applicable laws.

Credits

Original OBLITERATED model by OBLITERATUS
Quantization and tuning by wepiqx
Built with llama.cpp
Imatrix by mradermacher
Based on Google's Gemma 4 12B IT, released under Apache 2.0

More quantizations: wepiqx/gemma-4-12B-it-SHQ6-GGUF

License

This quantized model is distributed under the same Apache 2.0 license as the original Gemma 4 model.

Downloads last month: 222

GGUF

Model size

12B params

Architecture

gemma4

Hardware compatibility

4-bit

Model tree for wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF

Base model

google/gemma-4-12B

Finetuned

google/gemma-4-12B-it

Quantized

OBLITERATUS/Gemma-4-12B-OBLITERATED

Quantized

(9)

this model

wepiqx
/

gemma-4-12B-OBLITERATED-SHQ7-GGUF

Gemma 4 12B OBLITERATED -- SHQ7 (IQ-Hybrid)

Key Results

How We Made It

The Problem

Step 1: Q4_K_M baseline

Step 2: SHQ6 approach from QAT -- FAILED on OBLITERATED

Step 3: IQ-gradient -- CATASTROPHIC FAILURE

Step 4: Imatrix analysis

Step 5: IQ4XS_lean -- first success

Step 6: IQ4XS_lean_v2 -- TIED with Q4_K_M

Step 7: IQ4XS_lean_v3 -- THE WINNER

Step 8: Q6_K upgrade -- WORSE

What We Learned

About OBLITERATED

Architecture Notes

Weights

Repository Files

Quantization Command

All Experiments (OBLITERATED)

Hardware Requirements

Usage

llama.cpp

Ollama

LM Studio

Performance

Coding Example

Disclaimer

Credits

License

Model tree for wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF

Space using wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF 1