--- license: apache-2.0 base_model: llava-hf/llava-onevision-qwen2-7b-si-hf datasets: - aacudad/AnomalyThink library_name: transformers pipeline_tag: image-text-to-text tags: - anomaly-detection - industrial-anomaly-detection - vision-language-model - reasoning - llava_onevision --- # AnomalyThink-LLaVA-OneVision-7B-KCR The strongest LLaVA-OneVision model in this family. Trained with SFT only, on a self-distilled corpus that the model family generated itself. A **LLaVA-OneVision-7B-SI** model fine-tuned for **explainable industrial anomaly detection (IAD)**. Given one product image it writes a reasoning trace in ``, then a defect `` and `` when it sees a defect, and finally a binary ``. Research artefact from the MSc thesis *Reasoning-Enhanced Vision-Language Models for Explainable Industrial Anomaly Detection* (TU Delft, 2026). This is the cross-architecture replication of the thesis pipeline. The backbone is the same one IAD-R1 uses, so the comparison is on IAD-R1's own ground. ## Results Metric is **balanced accuracy**, `(TPR + TNR) / 2`, in percent. | Benchmark | Balanced accuracy | |---|---| | MMAD DS-MVTec (1,670 images) | **88.45** | | MMAD VisA (2,141 images) | **74.25** | **Which build these weights are.** This checkpoint is the *first* KCR build on this backbone. Its corpus was drawn from the whole 10,236-image rollout pool rather than from the 6,000-image SFT split (only 3,516 of its images lie inside that split, the other 2,484 come from the GRPO split) and it was balanced on the folder name rather than on the trace verdict, which left it at 45.0 % anomalous. The thesis reports the *corrected* build, rebuilt with all 6,000 images inside the SFT split and exactly 3,000 / 3,000 by verdict, as the replication result: 84.35 / 71.36, 87.32 / 72.65, 86.96 / 73.57 and 86.60 / 74.29 at epochs 1 to 4, with 87.32 / 72.65 the DS-MVTec-selected row. The first build stays on record in the thesis (Table 6.16, Appendix M). Both corpora are in the dataset repo: `llava_kcr/sft_llava_C_train.json` (first build, this model) and `llava_kcr/sft_llava_C_corrected_train.json` (corrected build). Reference rows measured on the exact same harness: | Model | DS-MVTec | VisA | |---|---|---| | LLaVA-OneVision-7B-SI base | 75.66 | 53.80 | | IAD-R1 released checkpoint | 81.92 | 71.34 | | AnomalyThink LLaVA SFT (6K Gemini traces) | 85.91 | 68.26 | | AnomalyThink LLaVA SFT then GRPO | 87.66 | 72.38 | | **This model (KCR, SFT only)** | **88.45** | **74.25** | **Evaluation protocol.** One shared harness for every row above. The DS-MVTec and VisA subsets of MMAD, single image per prompt, the same instruction the model was trained on, greedy decoding at temperature 0, at most 1024 new tokens, images capped at 262,144 pixels. Answers are parsed from the `` tag. Generation ran through vLLM 0.10.2, which agreed with the plain transformers generate path on 99 percent of a probe set. Nothing here is a re-scored or best-of-N number. Strict scoring: a generation with no parsable verdict counts as wrong, as in the thesis. Six of the 2,141 VisA generations of the SFT then GRPO model have no verdict, which gives 72.38; over the 2,135 parsable outputs it is 72.58. No ordering changes. ## Contamination note, please read this before you compare DS-MVTec numbers The public LLaVA-OneVision training mixture (`lmms-lab/LLaVA-OneVision-Data`, config `vision_flan(filtered)`) contains 426 rows whose id matches `%MVTecAD%`. The base model has therefore seen MVTec-AD material during its own instruction tuning. **Every DS-MVTec number for any LLaVA-OneVision derived model carries that caveat, including the 88.45 above, and including the IAD-R1 row.** We do not know how much of the gap is real capability and how much is recall. VisA is not affected. The same query over the mixture returns 0 rows for VisA. So the **74.25 on VisA is the clean number** and it is the one to trust for a cross-model comparison. ## Training * **Base model**: [`llava-hf/llava-onevision-qwen2-7b-si-hf`](https://huggingface.co/llava-hf/llava-onevision-qwen2-7b-si-hf). * **Corpus**: [`llava_kcr/sft_llava_C_train.json`](https://huggingface.co/datasets/aacudad/AnomalyThink/tree/main/llava_kcr), 6,000 traces, balanced 50/50 over normal and anomalous parts, on Real-IAD images. * **Recipe**: supervised fine-tuning only. No reinforcement learning stage. The SigLIP vision tower is frozen, the multimodal projector and the language model are trained. Learning rate 1e-5, cosine schedule, warmup ratio 0.03, weight decay 0.1, effective batch size 32, context cutoff 8,192 tokens, bf16, 4 epochs. * **Epoch**: this is **epoch 4 of 4** (step 748), the best of the four saved epochs. The full epoch curve on DS-MVTec / VisA was 85.29 / 72.32, 85.60 / 70.10, 86.45 / 73.56, and 88.45 / 74.25. ## The KCR corpus here is LLaVA native, not borrowed from Qwen This matters, so it is stated plainly. KCR stands for keep, correct, rewrite. The corpus for this model was built from **LLaVA-OneVision's own rollouts**, not from the Qwen rollouts used elsewhere in the thesis. The loop was: 1. Sample 10,236 rollouts on Real-IAD training images from the sibling [SFT then GRPO checkpoint](https://huggingface.co/aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO), k = 8 per image at temperature 0.7. 2. Bucket each trace by whether its verdict matched the label, into keep, needs correction, or needs rewrite. 3. Score every trace with a Gemini-3-Flash faithfulness judge, and demote traces that were right for the wrong reason. 4. Have Gemini-3-Flash correct or rewrite the traces that failed, then sample a balanced 6,000. The intermediate arms are published next to the final one: [`llava_kcr/`](https://huggingface.co/datasets/aacudad/AnomalyThink/tree/main/llava_kcr) holds `sft_llava_A_kept.json` (keep only), `sft_llava_B_kept_corrected.json` (keep plus corrected), `sft_llava_C_train.json` (the first-build corpus this model was trained on) and `sft_llava_C_corrected_train.json` (the corrected build the thesis reports, see the build note above). Because the rollouts come from this backbone, the corpus is on-policy for LLaVA-OneVision, and that is worth the extra loop. We also ran the control, which is the Qwen derived KCR corpus fine-tuned on this same LLaVA backbone with the same recipe. Its four epochs scored 87.70 / 70.97, 85.05 / 73.52, 86.97 / 69.95, and 85.85 / 70.75. No single checkpoint of that control reaches the 88.45 / 74.25 of the native corpus, and no checkpoint is best on both benchmarks at once. Building the corpus from the backbone's own rollouts is the difference. ## Tested transformers versions The weights and configs in this repo were written by **transformers 4.51.3**. Loading, the processor, and the full evaluation were verified under **transformers 4.57.1**, both through the plain HF generate path and through **vLLM 0.10.2**. One warning for anyone rebuilding this pipeline. Checkpoints saved by **transformers 5.0** write the rope settings under `text_config.rope_parameters`. Transformers 4.x does not read that key and silently falls back to `rope_theta = 10000`, which is 100 times too small. The model then stays fluent but goes blind and answers "no" to nearly everything, which looks like a collapsed run rather than a loading bug. This repo ships a plain `text_config.rope_theta = 1000000.0` next to the 5.0 style block, so it loads correctly on both major versions. ## Usage ```python import torch from PIL import Image from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration repo = "aacudad/AnomalyThink-LLaVA-OneVision-7B-KCR" model = LlavaOnevisionForConditionalGeneration.from_pretrained( repo, torch_dtype=torch.bfloat16, device_map="auto") processor = AutoProcessor.from_pretrained(repo) image = Image.open("part.png").convert("RGB") product = "tile" # the product category, it goes into the prompt question = ( f"Analyze the provided image of the {product}. " "Determine if there are any anomalies present. " "If an anomaly is detected, specify its type and location, " "and provide a detailed reasoning for your conclusion." ) messages = [{"role": "user", "content": [ {"type": "image"}, {"type": "text", "text": question}, ]}] prompt = processor.apply_chat_template(messages, add_generation_prompt=True) inputs = processor(images=image, text=prompt, return_tensors="pt").to( model.device, torch.bfloat16) out = model.generate(**inputs, max_new_tokens=1024, do_sample=False) print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) ``` Use this exact instruction. The model was trained on it and it degrades on a different phrasing. Expected output on a defective part: ``` I am inspecting a tile with a speckled, grayish-white surface. ... In the center of the tile, I detect a triangular, translucent plastic fragment. ... center Contamination Yes ``` On a normal part the model emits `` and then `No`, with no `` or `` tag. ## Intended use and limitations Research on explainable industrial anomaly detection. This is a thesis artefact, not a production inspection system. Known limitations: * The DS-MVTec contamination caveat above. * The model can write a confident and well argued trace for a defect that is not there. * The `` label is coarse and the model over-uses "missing parts" for any loss of material, including chips and gouges. * It was trained on Real-IAD style single-object images on plain backgrounds. Cluttered scenes, multiple parts per image, and very different lighting are out of distribution. * Reasoning traces were distilled from a teacher model. A fluent trace is not proof that the model looked at the right pixels. ## Citation ```bibtex @mastersthesis{acudad2026reasoning, author = {Acudad, A.}, title = {Reasoning-Enhanced Vision-Language Models for Explainable Industrial Anomaly Detection}, school = {Delft University of Technology}, year = {2026}, type = {Master's thesis}, url = {https://resolver.tudelft.nl/uuid:65c62420-79c0-447f-b095-7fb11d4474fc} } ``` Thesis: . Code and evaluation files: . The training data is at [aacudad/AnomalyThink](https://huggingface.co/datasets/aacudad/AnomalyThink). ## Related models * [AnomalyThink-LLaVA-OneVision-7B-SFT](https://huggingface.co/aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT) * [AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO](https://huggingface.co/aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO) * [AnomalyThink-Qwen2.5-VL-7B-KCR](https://huggingface.co/aacudad/AnomalyThink-Qwen2.5-VL-7B-KCR), the same pipeline on the Qwen2.5-VL backbone ## License Apache-2.0, inherited from the LLaVA-OneVision-7B-SI base. Trained on Real-IAD images, which are not redistributed here, so cite Real-IAD separately. Reasoning traces were distilled from Gemini 2.5-Flash.