---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- vision-language-model
- multimodal
- minicpm5
- siglip2
- unofficial
- experimental
base_model:
- openbmb/MiniCPM5-1B
- google/siglip2-base-patch16-512
---

# MiniCPM5-V-1B (unofficial, experimental)

**This is an unofficial, community/personal experiment. It is not affiliated with, endorsed by, or
produced by OpenBMB.** OpenBMB has never released a vision-capable version of MiniCPM5-1B (their
`MiniCPM-V` vision line is built on a different backbone, Qwen3.5, not MiniCPM5). This repo grafts
a vision encoder onto MiniCPM5-1B and trains a connector from scratch, because no such model
existed anywhere.

## What this is

- **Base LLM**: [`openbmb/MiniCPM5-1B`](https://huggingface.co/openbmb/MiniCPM5-1B) (apache-2.0),
  weights unmodified.
- **Vision encoder**: [`google/siglip2-base-patch16-512`](https://huggingface.co/google/siglip2-base-patch16-512),
  weights unmodified, frozen throughout training.
- **Connector**: a single linear projector (pixel-shuffle x4 + `Linear(12288, 1536)`, ~18.9M
  params) trained from random initialization. This is the only thing actually trained here.
- **Code**: the vision encoder loading/wiring and pixel-shuffle projector structure are adapted
  from [nanoVLM](https://github.com/huggingface/nanoVLM) (MIT license) — reimplemented, not copied
  verbatim, to plug into MiniCPM5's `transformers` `LlamaForCausalLM` interface instead of
  nanoVLM's own SmolLM2-based decoder.

## What this is NOT

- **Not a finished VLM.** This is a single stage-1 (LLaVA-style) alignment run: 10,000
  image-caption pairs from Flickr30k, 1 epoch (1,250 steps), batch size 8, projector-only
  gradient (both the vision encoder and the LLM stay frozen). Real VLM releases (including
  OpenBMB's own MiniCPM-V line) train on millions of image-text pairs across multiple stages.
  This model has seen roughly 0.1-1% as much data.
- **Not benchmarked.** No BFCL/MME/OCRBench/etc. numbers are reported here, because none were run.
  Anyone citing accuracy numbers for this model is making them up — don't.
- **Not reliably accurate.** See examples below: it correctly identifies broad scene content
  (people, objects, rough activity) at a noticeably-better-than-random rate, but regularly gets
  specific details wrong (color, exact object, count).

## Training details

- Data: 10,000 (image, caption) pairs, [`nlphuji/flickr30k`](https://huggingface.co/datasets/nlphuji/flickr30k)
  test split, no data augmentation.
- Optimizer: AdamW, lr 1e-3, linear warmup + linear decay, batch size 8, 1 epoch (1,250 steps).
- Only the projector's ~18.9M parameters received gradient updates. SigLIP2 (~86.4M params) and
  MiniCPM5-1B (~1.1B params) were frozen for the entire run.
- Loss (next-token cross-entropy on caption tokens only): started at 7.95 (first 20-step avg),
  ended at 2.78 (last 20-step avg) — a real, monotonic-ish decrease, not noise. Still slowly
  decreasing at the end of the run, meaning more training would likely help further; this is an
  early checkpoint, not a converged one.
- Hardware: single Colab T4 GPU, ~48 minutes wall-clock.

## Qualitative examples (held-out images, not seen during training)

| Ground-truth caption | Model output |
|---|---|
| "A young woman lifts her voice in song in a field near an old wooden structure." | "A girl wearing a red shirt and jeans is playing a guitar." |
| "A tan dog climbing over a bare tree trunk with a man standing in the background" | "A man in a red jacket and brown hair is standing on a tree with a dog." |
| "This is a lady in a blue outfit... holding a tennis racket in her hand." | "A woman wearing a red dress... is holding a tennis ball." |
| "Young boy in a white striped shirt and headband holding a tennis racket." | "A young boy wearing a white shirt and shorts is playing a game with a ball." |

The model picks up real scene elements (dog + tree + man; tennis-adjacent activity + ball/racket)
but gets specifics wrong (colors, exact object). This is the expected shape of output for a
lightly-trained stage-1 projector, not a bug.

## Usage

Requires the base MiniCPM5-1B and SigLIP2 weights (downloaded automatically via `transformers`/
`huggingface_hub`) plus the small `model.py` / `vision_encoder.py` wrapper and this repo's trained
`projector.safetensors`.

```python
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

# model.py / vision_encoder.py from this repo define MiniCPM5V
from model import MiniCPM5V

model = MiniCPM5V().to("cuda")
proj_path = hf_hub_download("ewinregirgojr/MiniCPM5-V-1B-unofficial", "projector.safetensors")
model.projector.load_state_dict(load_file(proj_path))
model.eval()

from PIL import Image
import numpy as np

def img_transform(img):
    img = img.resize((512, 512), Image.BICUBIC)
    arr = (np.asarray(img, dtype=np.float32) / 255.0 - 0.5) / 0.5
    return torch.from_numpy(arr).permute(2, 0, 1).contiguous()

image = Image.open("your_image.jpg").convert("RGB")
pixel_values = img_transform(image).unsqueeze(0).to("cuda", dtype=torch.bfloat16)
ids = model.make_prompt_ids("Describe the image.\n", "cuda")
attn = torch.ones_like(ids)
out = model.generate(input_ids=ids, attention_mask=attn, pixel_values=pixel_values, max_new_tokens=40)
print(model.tokenizer.decode(out[0], skip_special_tokens=True))
```

## Limitations

- English captioning only tested; no multilingual, multi-image, video, or OCR capability trained.
- Single-image only.
- No safety/alignment tuning specific to vision inputs.
- Small stage-1 run — expect hallucination and detail errors, not a reliable VQA/captioning tool.

## License

MiniCPM5-1B and SigLIP2 weights are used unmodified under their original licenses (both
apache-2.0). This repo's own contribution (the projector weights and wrapper code) is released
under apache-2.0. Code structure for the vision encoder/projector is adapted from nanoVLM
(MIT license, https://github.com/huggingface/nanoVLM) — changes: reimplemented to target
MiniCPM5-1B's `transformers` interface instead of nanoVLM's native decoder, image-token handling
adjusted to MiniCPM5's tokenizer/vocab.

Not affiliated with or endorsed by OpenBMB, Google, or the nanoVLM authors.