ruisv/bllm-qwen3.5-4b

Qwen3.5-4B — BPU-native build for D-Robotics RDK S100P

Compiled, quantised (int8 weights / int16 per-linear-calibrated activations), and board-verified build of Qwen/Qwen3.5-4B for D-Robotics' Nash BPU, running entirely through BLLM — an on-device runtime that talks to the BPU directly (hbDNN/hbUCP), with no GPU/CUDA and no cloud round-trip.

This is a compiled BPU binary, not a transformers-loadable checkpoint — closer in kind to a GGUF quantisation than a HF AutoModel. There is no Inference API widget on this page; see "How to run" below.

New here? Start with BLLM (the runtime) and bllm-model-zoo (how this build was made, full acceptance data, what was tried and rejected). This repo is the downloadable artifact; that repo is the method.

Quick facts

Base model	Qwen/Qwen3.5-4B (Apache-2.0)
Architecture	Hybrid: 24 Gated-DeltaNet layers (decoupled 16 key / 32 value heads) + 8 full-attention layers
Quantisation	int8 weight (per-channel) / int16 activation, per-linear calibrated (flat ranges clip this size — see below)
Target hardware	D-Robotics RDK S100P — Journey-6 J6M, compiled march nash-m. Not the 80-TOPS RDK S100 (J6E, nash-e), which needs its own build: at this size none is published, see the 0.8B card for an S100 build
Modality	Image + text (vision tower included)
BPU utilization	100% — CPU_inference_time_cost == 0 on the text decoder
Runtime	BLLM (native, no libxlm)

This architecture (linear-attention SSM + head_dim=256 + decoupled GDN key/value head counts) is not covered by D-Robotics' official OE-LLM toolchain (libxlm) — this build exists because BLLM's native runtime compiles the graph directly.

Files in this repo

s100p/
  ctx4k/    # cache_len=4096 — default, recommended context
    model.hbm            # text decoder (prefill+decode dual-graph)
    model_prefill.hbm     # chunked-prefill graph (N=32)
    model.json
    embed_tokens.bin
    tokenizer.json
    visual.hbm            # vision tower, 320px bucket, 100 tokens/image
  ctx512/   # cache_len=512 — faster decode, short-context use
    ...same layout

cache_len is a performance knob as well as a capacity one — a larger KV window costs more DDR traffic per decoded token. Pick ctx512 if your prompts/replies genuinely fit in ~512 tokens; it decodes faster than ctx4k.

How to run

conda install -c https://mirrors.ruis.ai/conda -c conda-forge bllm

import bllm
m = bllm.load("s100p/ctx4k")
print(m.chat("这张图里有什么?", images=["photo.jpg"]))

# text-only, without loading the vision tower into memory
m = bllm.load("s100p/ctx4k", vision=False)
print(m.chat("介绍一下你自己"))

Requires a D-Robotics RDK S100P (an RDK S100 is a different BPU tier and will not load these files — see "Target hardware" above), sw >= 4.0.5, and an enlarged ION carveout: hb_switch_ion.sh bpu_first + reboot. balanced is too small at this size — its 7.5 GB pool cannot hold the two text graphs (4.27 GB + 4.52 GB). See BLLM's docs/LLM_ONBOARD.md.

Update 2026-11 — a short prompt reaches its first token 5.5–6.8x sooner

model_prefill.hbm and model.json were replaced in every folder; every other file is unchanged (the previous files are at revision b5370786ad7e).

A prompt is prefilled 32 tokens at a time. The part that did not fill a whole chunk — all of a typical question — used to go through the decode graph one token per step, and every step reads all of the model's weights from memory once, while one prefill chunk reads them once for 32 tokens. So time to first token was a cliff rather than a slope: a 31-token prompt took several times longer than a 32-token one. The new prefill graph takes that tail as one padded chunk: the pad rows leave the recurrent state untouched and the runtime takes their K/V back out of the attention window. Tails shorter than prefill_min_tail tokens (measured per folder, where stepping is still faster) are stepped as before.

folder	31-token prompt, before → after	prefill_min_tail	whole 32-token chunk (graph)	next-token top-1, padded vs stepped	first reply token, 12 chats
s100p/ctx4k	5.09 s → 0.75 s	5	739 → 755 ms	64/64	12/12
s100p/ctx512	4.24 s → 0.78 s	6	658 → 673 ms	63/64	11/12*

Measured on an RDK S100P, performance mode, feeding a text prompt up to the first token's logits. Wherever a chunk is full, the new graph gives bit-identical caches and logits to the previous one (every cache compared on its real values after 32 and 64 tokens). A padded tail and a stepped one are the same computation on two different int8 graphs (prefill vs decode), so they agree up to those graphs' numerics: next-token perplexity over 64 prompts whose tails cover every length (each target's log-probability read from the top 1000) moves by -0.06% to +0.20%, and prefill↔decode parity still passes in every folder.

* One of the 12 chat prompts ("1+1等于几？") is a near-tie on its first token: "1" 0.502 vs "在" 0.489 stepped, "在" 0.500 vs "1" 0.491 padded. Both replies are correct; a padded and a stepped tail run on two different int8 graphs, so a tie this close can flip.

Which bllm. The new graph has the same inputs and outputs as the old one, so an older bllm (≤ 0.7.0) runs these files exactly as it ran the previous ones — same speed, same answers. bllm 0.8.0 reads "prefill_pads": true from model.json and pads the tail; model.json also names the chat template family ("dialect": "qwen3.5"), which 0.8.0's OpenAI server uses for tool calls and reasoning_content.

To update: conda install -c https://mirrors.ruis.ai/conda -c conda-forge bllm=0.8.0, and hf download your folder again — only these two files changed.

Measured on hardware (S100P, performance mode, 3 repeated runs)

variant	decode	TTFT (image, coco-cats test photo)	prefill↔decode parity	ppl	image contrast	task anchors
ctx4096	6.14 tok/s	3.29 s	PASS	35.29	PASS	PASS
ctx512	7.54 tok/s	2.93 s	PASS	35.09	PASS	PASS

Decode is nearly 2x the pre-rewrite baseline (3.29 tok/s) — see "Known limitations" below for what changed and why that isn't a free lunch. Full methodology, acceptance protocol (prefill↔decode parity + perplexity + task-coherence anchors + with/without-image contrast — not just cosine similarity, see why in the model-zoo README), and every rejected build are in bllm-model-zoo/models/qwen3.5-4b and .../qwen3.5-vlm.

Known limitations

• This release changed 4B's attention implementation from an expand_win broadcast to GQA batched matmul + mask deduplication (the same rewrite applied to the 0.8B/2B siblings) — this is where the ~2x decode speedup above comes from (most of expand_win's redundant DMA is gone). Unlike the 0.8B/2B siblings, this introduces new int16 quantisation on the attention score/output matmul that the prior expand_win path never had — board output after the rewrite is not byte-identical to the pre-rewrite build (still coherent and correct, e.g. one prior answer mentioned an extra detail the new build's phrasing omits). The batched-matmul math itself was checked algebraically exact against the old math offline (cosine 1.0000001, max abs diff 2.38e-7 on this model's exact head configuration), and this release's full acceptance run (table above) is what actually clears the build for use — the offline math check alone is not a substitute for it.
• Flat (uncalibrated) int16 activation ranges — which fit the 0.8B/2B siblings — clip this size's deep linears into a degenerate loop; this build uses per-linear calibration instead (see the model-zoo recipe for the measured activation ranges and why).
• The all-BPU chunked-prefill path (image ingestion, no CPU fallback) is numerically solvable but impractically slow to compile — this build's prefill graph uses a float [C,C] CPU fallback for one small intermediate instead.

License

The compiled .hbm is a derivative of the upstream Qwen3.5-4B weights and inherits its Apache-2.0 license — check the upstream model card for the current terms. BLLM (the runtime that loads this file) is open source; see its repository for its own license.