---
library_name: transformers
tags:
  - medical
  - activation-steering
  - gated-activation-steering
  - inference-time-intervention
  - sycophancy
  - hallucination
---

# Gated Activation Steering for Reducing Sycophancy & Hallucination in Medical Question Answering

**Inference-time steering for MedGemma that improves grounding in the patient record and resistance to user pressure, without target-model fine-tuning or weight updates.** No fine-tuning or model-weight changes are performed; instead, a small, gated intervention is applied to selected attention heads only during inference while the model generates its response.

📄 **Paper:** [Gated Activation Steering for Reducing Sycophancy & Hallucination in Medical Question Answering](https://arxiv.org/abs/2608.23666)


> ⚕️ **Research artifact — not a medical device.** For research use only. Not for clinical
> decision-making, diagnosis, or treatment.

This repository ships **only the steering artifacts and inference-time runtime code** — it does **not** contain MedGemma weights.
You download MedGemma from Google and run these on top of it.

---

## TL;DR

- **What it does:** two independent, gated interventions — one that reduces **hallucination**
  (rejecting claims the EHR does not support) and one that reduces **sycophancy** (not caving when
  a user pushes a false value). Each fires *only when needed*, so ordinary correct answers are left
  untouched.
- **How:** Inference-Time Intervention (ITI). Separate steering directions are learned from
  contrastive clinical pairs and applied to **causally-verified attention heads**, gated at runtime
  by lightweight probes that set a continuous 0–1 strength per turn.

---

## Files

| File | Description |
|---|---|
| `iti_H.pt` | Hallucination heads: indices + per-head 256-d direction vectors + scales |
| `iti_S.pt` | Sycophancy heads: same structure |
| `claim_probe.json` | False-claim detector — sets the H gate/strength |
| `gate_probe.json` | User-pressure detector — sets the S gate/strength |
| `steering_config.json` | Calibrated strengths and schedule |
| `config.py`, `common.py`, `iti.py`, `compose.py` | Runtime loader + steering code (inference only) |
| `test.py` | Runnable demo (base vs steered on a sample EHR) |
| `NOTICE`, `GOOGLE-HAI-DEF-TERMS.txt` | MedGemma usage note + HAI-DEF terms pointer |

The `.pt` files are only a few KB each (a handful of unit vectors) — **no model weights**.

## Requirements

```bash
pip install -r requirements.txt   # torch, transformers, accelerate, safetensors, sentencepiece, numpy, tqdm
```

CPU or GPU is auto-detected (bf16 on GPU, fp32 on CPU). The 4B model runs on CPU but is slow.

## Get MedGemma (local — this repo never downloads it)

```bash
huggingface-cli download google/medgemma-1.5-4b-it --local-dir ./medgemma-1.5-4b-it
```

## Run

```bash
# point the code at your local MedGemma folder
export MEDGEMMA_PATH=./medgemma-1.5-4b-it            # Windows: $env:MEDGEMMA_PATH = "C:\...\medgemma-1.5-4b-it"
python test.py
```

`test.py` prints base vs steered answers for a seed question and four escalating pressure turns, and
writes `demo_output.json`. If `MEDGEMMA_PATH` is unset (or not a real folder) the script stops with
instructions — it will not fetch anything from the Hub.

---

## How it works

1. **Learn directions** from contrastive clinical pairs (grounded vs. caving completions) for each
   behavior — hallucination (H) and sycophancy (S).
2. **Localize + causally verify** the attention heads that carry each behavior; keep only heads that
   change the behavior when ablated. H and S land on disjoint heads.
3. **Gate at runtime.** Two probes read the current turn: a *claim* probe (is there an unsupported
   claim?) and a *pressure* probe (is the user pushing?). Each outputs a continuous strength in
   `[0, 1]`.
4. **Apply the nudge** per generated token to the selected heads: `alpha · strength · decay(t) ·
   sigma · direction`, residual-norm-capped so it never overwhelms the activation. A calm turn gets
   almost nothing; a strong pressure turn gets a firm push. Model weights stay frozen.

---

## MedGemma Base Model and Terms

The steering vectors, detector probes, and accompanying runtime code in this repository were
developed by the authors.

They are designed for use with `google/medgemma-1.5-4b-it`.

This repository does not contain or redistribute MedGemma model weights. Users must obtain MedGemma
separately from Google and comply with the Health AI Developer Foundations (HAI-DEF) Terms of Use
applicable to their use of MedGemma.

MedGemma can be obtained from:

https://huggingface.co/google/medgemma-1.5-4b-it

See `NOTICE` and `GOOGLE-HAI-DEF-TERMS.txt` for the applicable HAI-DEF information.