--- license: apache-2.0 base_model: Qwen/Qwen3.5-4B pipeline_tag: image-text-to-text tags: - bpu - d-robotics - rdk - edge - on-device - int8 - gated-deltanet - hybrid-ssm - vision-language language: - en - zh --- # Qwen3.5-4B — BPU-native build for D-Robotics RDK S100P Compiled, quantised (int8 weights / int16 per-linear-calibrated activations), and board-verified build of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) for D-Robotics' Nash BPU, running entirely through [BLLM](https://github.com/ruisv/bllm) — an on-device runtime that talks to the BPU directly (`hbDNN`/`hbUCP`), with no GPU/CUDA and no cloud round-trip. This is a **compiled BPU binary**, not a `transformers`-loadable checkpoint — closer in kind to a GGUF quantisation than a HF `AutoModel`. There is no Inference API widget on this page; see "How to run" below. > **New here?** Start with [BLLM](https://github.com/ruisv/bllm) (the runtime) > and [bllm-model-zoo](https://github.com/ruisv/bllm-model-zoo) (how this build > was made, full acceptance data, what was tried and rejected). This repo is > the downloadable artifact; that repo is the method. ## Quick facts | | | |---|---| | Base model | [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) (Apache-2.0) | | Architecture | Hybrid: 24 Gated-DeltaNet layers (**decoupled** 16 key / 32 value heads) + 8 full-attention layers | | Quantisation | int8 weight (per-channel) / int16 activation, **per-linear calibrated** (flat ranges clip this size — see below) | | Target hardware | D-Robotics RDK **S100P** — Journey-6 J6M, compiled `march nash-m`. **Not** the 80-TOPS RDK S100 (J6E, `nash-e`), which needs its own build: at this size none is published, see the [0.8B](https://huggingface.co/ruisv/bllm-qwen3.5-0.8b) card for an S100 build | | Modality | Image + text (vision tower included) | | BPU utilization | 100% — `CPU_inference_time_cost == 0` on the text decoder | | Runtime | [BLLM](https://github.com/ruisv/bllm) (native, no `libxlm`) | This architecture (linear-attention SSM + `head_dim=256` + decoupled GDN key/value head counts) is **not** covered by D-Robotics' official OE-LLM toolchain (`libxlm`) — this build exists because BLLM's native runtime compiles the graph directly. ## Files in this repo ``` s100p/ ctx4k/ # cache_len=4096 — default, recommended context model.hbm # text decoder (prefill+decode dual-graph) model_prefill.hbm # chunked-prefill graph (N=32) model.json embed_tokens.bin tokenizer.json visual.hbm # vision tower, 320px bucket, 100 tokens/image ctx512/ # cache_len=512 — faster decode, short-context use ...same layout ``` `cache_len` is a **performance** knob as well as a capacity one — a larger KV window costs more DDR traffic per decoded token. Pick `ctx512` if your prompts/replies genuinely fit in ~512 tokens; it decodes faster than `ctx4k`. ## How to run ```bash conda install -c https://mirrors.ruis.ai/conda -c conda-forge bllm ``` ```python import bllm m = bllm.load("s100p/ctx4k") print(m.chat("这张图里有什么?", images=["photo.jpg"])) # text-only, without loading the vision tower into memory m = bllm.load("s100p/ctx4k", vision=False) print(m.chat("介绍一下你自己")) ``` Requires a D-Robotics RDK S100P (an RDK S100 is a different BPU tier and will not load these files — see "Target hardware" above), `sw >= 4.0.5`, and an enlarged ION carveout: `hb_switch_ion.sh bpu_first` + reboot. `balanced` is too small at this size — its 7.5 GB pool cannot hold the two text graphs (4.27 GB + 4.52 GB). See BLLM's `docs/LLM_ONBOARD.md`. ## Update 2026-11 — a short prompt reaches its first token 5.5–6.8x sooner `model_prefill.hbm` and `model.json` were replaced in every folder; every other file is unchanged (the previous files are at revision `b5370786ad7e`). A prompt is prefilled 32 tokens at a time. The part that did not fill a whole chunk — all of a typical question — used to go through the decode graph one token per step, and every step reads all of the model's weights from memory once, while one prefill chunk reads them once for 32 tokens. So time to first token was a cliff rather than a slope: a 31-token prompt took several times longer than a 32-token one. The new prefill graph takes that tail as **one padded chunk**: the pad rows leave the recurrent state untouched and the runtime takes their K/V back out of the attention window. Tails shorter than `prefill_min_tail` tokens (measured per folder, where stepping is still faster) are stepped as before. | folder | 31-token prompt, before → after | `prefill_min_tail` | whole 32-token chunk (graph) | next-token top-1, padded vs stepped | first reply token, 12 chats | |---|---|---|---|---|---| | `s100p/ctx4k` | 5.09 s → 0.75 s | 5 | 739 → 755 ms | 64/64 | 12/12 | | `s100p/ctx512` | 4.24 s → 0.78 s | 6 | 658 → 673 ms | 63/64 | 11/12\* | Measured on an RDK S100P, performance mode, feeding a text prompt up to the first token's logits. Wherever a chunk is full, the new graph gives **bit-identical** caches and logits to the previous one (every cache compared on its real values after 32 and 64 tokens). A padded tail and a stepped one are the same computation on two different int8 graphs (prefill vs decode), so they agree up to those graphs' numerics: next-token perplexity over 64 prompts whose tails cover every length (each target's log-probability read from the top 1000) moves by -0.06% to +0.20%, and prefill↔decode parity still passes in every folder. \* One of the 12 chat prompts ("1+1等于几?") is a near-tie on its first token: "1" 0.502 vs "在" 0.489 stepped, "在" 0.500 vs "1" 0.491 padded. Both replies are correct; a padded and a stepped tail run on two different int8 graphs, so a tie this close can flip. **Which bllm.** The new graph has the same inputs and outputs as the old one, so an older bllm (≤ 0.7.0) runs these files exactly as it ran the previous ones — same speed, same answers. bllm **0.8.0** reads `"prefill_pads": true` from `model.json` and pads the tail; `model.json` also names the chat template family (`"dialect": "qwen3.5"`), which 0.8.0's OpenAI server uses for tool calls and `reasoning_content`. **To update**: `conda install -c https://mirrors.ruis.ai/conda -c conda-forge bllm=0.8.0`, and `hf download` your folder again — only these two files changed. ## Measured on hardware (S100P, performance mode, 3 repeated runs) | variant | decode | TTFT (image, coco-cats test photo) | prefill↔decode parity | ppl | image contrast | task anchors | |---|---|---|---|---|---|---| | ctx4096 | 6.14 tok/s | 3.29 s | PASS | 35.29 | PASS | PASS | | ctx512 | 7.54 tok/s | 2.93 s | PASS | 35.09 | PASS | PASS | **Decode is nearly 2x the pre-rewrite baseline** (3.29 tok/s) — see "Known limitations" below for what changed and why that isn't a free lunch. Full methodology, acceptance protocol (prefill↔decode parity + perplexity + task-coherence anchors + with/without-image contrast — **not just cosine similarity**, see why in the [model-zoo README](https://github.com/ruisv/bllm-model-zoo#the-acceptance-contract-why-it-is-not-cosine)), and every rejected build are in [bllm-model-zoo/models/qwen3.5-4b](https://github.com/ruisv/bllm-model-zoo/tree/main/models/qwen3.5-4b) and [.../qwen3.5-vlm](https://github.com/ruisv/bllm-model-zoo/tree/main/models/qwen3.5-vlm). ## Known limitations - **This release changed 4B's attention implementation** from an `expand_win` broadcast to GQA batched matmul + mask deduplication (the same rewrite applied to the 0.8B/2B siblings) — this is where the ~2x decode speedup above comes from (most of `expand_win`'s redundant DMA is gone). Unlike the 0.8B/2B siblings, this introduces new int16 quantisation on the attention score/output matmul that the prior `expand_win` path never had — board output after the rewrite is **not byte-identical** to the pre-rewrite build (still coherent and correct, e.g. one prior answer mentioned an extra detail the new build's phrasing omits). The batched-matmul math itself was checked algebraically exact against the old math offline (cosine 1.0000001, max abs diff 2.38e-7 on this model's exact head configuration), and this release's full acceptance run (table above) is what actually clears the build for use — the offline math check alone is not a substitute for it. - Flat (uncalibrated) int16 activation ranges — which fit the 0.8B/2B siblings — clip this size's deep linears into a degenerate loop; this build uses per-linear calibration instead (see the model-zoo recipe for the measured activation ranges and why). - The all-BPU chunked-prefill path (image ingestion, no CPU fallback) is numerically solvable but impractically slow to compile — this build's prefill graph uses a float `[C,C]` CPU fallback for one small intermediate instead. ## License The compiled `.hbm` is a derivative of the upstream [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) weights and inherits its **Apache-2.0** license — check the upstream model card for the current terms. BLLM (the runtime that loads this file) is open source; see [its repository](https://github.com/ruisv/bllm) for its own license.