---
license: other
license_name: lfm1.0
license_link: https://huggingface.co/LiquidAI/LFM2.5-VL-3B/blob/main/LICENSE
base_model: LiquidAI/LFM2.5-VL-3B
base_model_relation: quantized
pipeline_tag: image-text-to-text
library_name: transformers
tags:
  - awq
  - int4
  - w4a16
  - compressed-tensors
  - llm-compressor
language:
  - en
---

# LFM2.5-VL-3B-AWQ

AWQ (W4A16) quantization of
[LiquidAI/LFM2.5-VL-3B](https://huggingface.co/LiquidAI/LFM2.5-VL-3B)
— a ~3B vision-language model from Liquid AI (`lfm2_vl`, native in `transformers`): a SigLIP2 NaFlex 400M vision encoder + the LFM2.5-2.6B hybrid conv+attention language backbone. Only the LFM2 language backbone is quantized here (self-attention + feed-forward projections); the SigLIP2 vision encoder, the short-convolution operators, the vision→LM projector, and all norms/embeddings are kept in BF16.

**Variant**: AWQ **W4A16** — 4-bit symmetric integer weights, group size 128, with activation-aware scaling. Activations stay BF16.
**Quantized by**: [sahilchachra](https://huggingface.co/sahilchachra)
**Tooling**: `llm-compressor` (`AWQModifier` + `QuantizationModifier`) -> `compressed-tensors` `pack-quantized`

> This is a quantized derivative. Weights, behavior, and license follow the base
> model — see the
> [original card](https://huggingface.co/LiquidAI/LFM2.5-VL-3B) for full details, benchmarks, and citation.

## What is quantized

Quantized to 4-bit:

- LM backbone `model.language_model.layers.*.self_attn.{q,k,v,out}_proj` (8 full-attention layers)
- LM backbone `model.language_model.layers.*.feed_forward.{w1,w2,w3}` (all 30 layers; SwiGLU MLP)

Kept in **BF16**: SigLIP2 NaFlex vision encoder (`model.vision_tower.*`), LFM2 short-convolution operators (`model.language_model.layers.*.conv.*` — 22 conv layers), vision->LM projector (`model.multi_modal_projector.*`), token embeddings, lm_head, all RMSNorms (operator_norm / ffn_norm / q_layernorm / k_layernorm).

### Runtime

Load with `transformers>=5.0.0` (the `lfm2_vl` architecture is native — no `trust_remote_code` needed) using `AutoModelForImageTextToText` + `AutoProcessor`. The AWQ checkpoint is a compressed-tensors W4A16 (INT4, group size 128) model over the LFM2 language backbone; on-the-fly int4 dequant happens inside `transformers`.

### What is and isn't quantized

This is a *language-backbone-only* AWQ. The SigLIP2 vision encoder, the 22 short-convolution operators of the hybrid LFM2 backbone, the vision->LM projector, and all norms/embeddings stay in BF16 — only the 8 full-attention layers' q/k/v/out projections and every layer's SwiGLU feed-forward (w1/w2/w3) are 4-bit. This keeps image understanding and the conv mixing path at full precision while shrinking the bulk of the LM weights.

## Calibration

AWQ: 128 real image+text VQA samples streamed from HuggingFaceM4/the_cauldron (vqav2 config), each rendered through the model's own chat template + AutoProcessor so the LM backbone is calibrated on the true image-token activation distribution it sees at inference. Activation-aware scales use custom per-layer-anchored AWQ mappings (LFM2 uses operator_norm/ffn_norm rather than input_layernorm/post_attention_layernorm, and only the 8 full-attention layers carry q/k/v).

## Prompt template & sampling

LFM2.5-VL uses a ChatML-like template. Load with `transformers` (native `lfm2_vl`, requires `transformers>=5.0.0`) via `AutoModelForImageTextToText` + `AutoProcessor`. Pass OpenAI-style messages with image content, e.g. `messages=[{"role":"user","content":[{"type":"image","url":"..."},{"type":"text","text":"Describe this image."}]}]`, and render with `processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", tokenize=True, return_dict=True)`. Do NOT insert `<image>` tags yourself — `apply_chat_template` inserts them automatically per image. Liquid recommends `temperature=0.2`, `top_k=50`, `repetition_penalty=1.0` for text generation.

**Recommended sampling**: `temperature=0.2`, `top_k=50`, `repetition_penalty=1.0` (per the Liquid AI model card); `max_new_tokens` per use case.


## Verification

Validated end-to-end on **NVIDIA Jetson Thor** (Blackwell, `aarch64`, 128 GB unified memory), CUDA 13.0, PyTorch 2.13.0, `transformers` 5.15.1, `compressed-tensors` 0.17.0.

**Loaded with** (native `lfm2_vl` — no `trust_remote_code`; requires `transformers>=5.0.0`):

```python
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

proc = AutoProcessor.from_pretrained("sahilchachra/LFM2.5-VL-3B-AWQ")
model = AutoModelForImageTextToText.from_pretrained(
    "sahilchachra/LFM2.5-VL-3B-AWQ", dtype=torch.bfloat16, device_map="cuda:0").eval()
```

**Sanity check** — real VQAv2 images, greedy decoding (`do_sample=False`):

| Question | AWQ answer | Reference |
| --- | --- | --- |
| Is this a clear day? | No. | No. |
| Is this room tidy? | Yes. | Yes. |
| Is this a picture of a studio apartment? | Yes. | Yes. |
| What is the dog riding on? | Surfboard. | Surfboard. |

Free-form (`"Describe this image in detail."`): *“a small black and tan dog is seen riding a white surfboard on a calm, blue ocean…”*

**Decode speed**: ~25.7 tok/s (single image, 128–256 new tokens, greedy) on one Thor GPU. Shapes verified: 122/122 quantized linears group-128 consistent; output logits `(1, seq, 128000)`; SigLIP2 vision tower kept dense BF16.

Also verified serving under **vLLM 0.27.1** (same answers, ~62 tok/s decode) — see [Usage (vLLM)](#usage-vllm) below.

## Usage (vLLM)

Verified serving on **vLLM 0.27.1** — the `lfm2_vl` architecture is native (**no `trust_remote_code`**) and the `compressed-tensors` W4A16 quantization is auto-detected. Requires a vLLM build with `transformers>=5.0` (e.g. **≥ v0.27.1**); older builds that pin `transformers<5.0` cannot load the LFM2.5 tokenizer.

### Offline (multimodal)

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="sahilchachra/LFM2.5-VL-3B-AWQ",
    max_model_len=8192,                 # raise toward the base model's context as needed
    limit_mm_per_prompt={"image": 1},
)
messages = [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "https://path/to/image.jpg"}},
    {"type": "text", "text": "Describe this image in detail."},
]}]
out = llm.chat(
    messages,
    SamplingParams(temperature=0.2, top_k=50, repetition_penalty=1.0, max_tokens=512),
)
print(out[0].outputs[0].text)
```

### Server (OpenAI-compatible)

```bash
vllm serve sahilchachra/LFM2.5-VL-3B-AWQ \
    --max-model-len 8192 \
    --limit-mm-per-prompt '{"image": 1}'
```

Then send an `image_url` content part to `/v1/chat/completions` as usual.

**Measured on one NVIDIA Jetson Thor GPU**: loads in ~4 s (3.0 GiB weights), ~62 tok/s decode (single image, 128 new tokens, greedy).