Medical-NER-Qwen-4B-Thinking-GRPO-GGUF

Medical Named Entity Recognition (NER) model in GGUF format for use with llama.cpp, Ollama, and other GGUF-compatible inference engines.

Model Description

This model is fine-tuned from Qwen3-4B-Thinking-2507 using GRPO (Group Relative Policy Optimization) for medical literature entity extraction.

Capabilities

• Entity Recognition: Virus, Bacteria, Fungus, Parasite, Prion, Disease, Evidence
• Relationship Extraction: is_a, biomarker_for, correlated_with, has_relationship
• Output Format: Structured JSON

Available Quantizations

File	Size	Description
medical-ner-qwen-4b-thinking-f16.gguf	~8GB	Full precision (FP16)
medical-ner-qwen-4b-thinking-q8_0.gguf	~4GB	8-bit quantization (recommended)
medical-ner-qwen-4b-thinking-q4_k_m.gguf	~2.5GB	4-bit quantization (balanced)

Usage

With llama.cpp

./llama-cli -m medical-ner-qwen-4b-thinking-q8_0.gguf \
    -p "<|im_start|>system
You are an expert medical literature analyst...<|im_end|>
<|im_start|>user
Analyze: Epstein-Barr virus is associated with multiple sclerosis.<|im_end|>
<|im_start|>assistant
"

With Ollama

# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./medical-ner-qwen-4b-thinking-q8_0.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"""

PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.7
EOF

# Import and run
ollama create medical-ner -f Modelfile
ollama run medical-ner "Analyze: HIV causes AIDS."

Expected Output

{
  "entities": [
    {"text": "Epstein-Barr virus", "label": "Virus"},
    {"text": "EBV", "label": "Virus"},
    {"text": "multiple sclerosis", "label": "Disease"},
    {"text": "MS", "label": "Disease"}
  ],
  "relations": [
    {
      "head": "Epstein-Barr virus",
      "tail": "multiple sclerosis", 
      "relation": "correlated_with"
    }
  ]
}

Training Details

• Base Model: Qwen3-4B-Thinking-2507
• Method: SFT + GRPO (Group Relative Policy Optimization)
• Dataset: Medical literature with pathogen-disease relationships
• Training Samples: 1,557

Related Models

• LoRA Adapters: xingqiang/Medical-NER-Qwen-4B-Thinking-GRPO
• Merged 16bit: xingqiang/Medical-NER-Qwen-4B-Thinking-GRPO-merged-16bit

License

Apache 2.0