ruisv/bllm-qwen3.5-2b

Qwen3.5-2B — BPU-native build for D-Robotics RDK S100 / S100P

Compiled, quantised (int8 weights / int16 activations), and board-verified build of Qwen/Qwen3.5-2B for D-Robotics' Nash BPU, running entirely through BLLM — an on-device runtime that talks to the BPU directly (hbDNN/hbUCP), with no GPU/CUDA and no cloud round-trip.

This is a compiled BPU binary, not a transformers-loadable checkpoint — closer in kind to a GGUF quantisation than a HF AutoModel. There is no Inference API widget on this page; see "How to run" below.

New here? Start with BLLM (the runtime) and bllm-model-zoo (how this build was made, full acceptance data, what was tried and rejected). This repo is the downloadable artifact; that repo is the method.

Quick facts

Base model	Qwen/Qwen3.5-2B (Apache-2.0)
Architecture	Hybrid: 18 Gated-DeltaNet (linear-attention SSM) layers + 6 full-attention layers
Quantisation	int8 weight (per-channel) / int16 activation (static, data-free)
Target hardware	D-Robotics RDK S100 (s100/, nash-e) and S100P (s100p/, nash-m) — two separate builds, see below
Modality	Image + text (vision tower included)
BPU utilization	100% — CPU_inference_time_cost == 0 on the text decoder
Runtime	BLLM (native, no libxlm)

This architecture (linear-attention SSM + head_dim=256) is not covered by D-Robotics' official OE-LLM toolchain (libxlm) — this build exists because BLLM's native runtime compiles the graph directly.

Files in this repo

s100p/        # RDK S100P (Journey-6 J6M, 128 TOPS) — compiled march nash-m
  ctx4k/      # cache_len=4096 — default, recommended context
    model.hbm            # text decoder (prefill+decode dual-graph)
    model_prefill.hbm     # chunked-prefill graph (N=32)
    model.json
    embed_tokens.bin
    tokenizer.json
    visual.hbm            # vision tower, 320px bucket, 100 tokens/image
  ctx512/     # cache_len=512 — faster decode, short-context use
    ...same layout
s100/         # RDK S100  (Journey-6 J6E, 80 TOPS)  — compiled march nash-e
  ctx4k/      # same six files, same graph and quantisation, different BPU target
  ctx512/

Which folder do I need?

A BPU march names a Journey-6 SoC tier, not a board family, and an .hbm loads only on its own tier:

march	SoC	board
nash-e	J6E, 80 TOPS, 1 BPU core	RDK S100 (12 GB)
nash-m	J6M, 128 TOPS, 1 BPU core	RDK S100P (24 GB)
nash-p	J6P, 4 BPU cores	RDK S600

Take s100/ for an RDK S100 and s100p/ for an RDK S100P. On an S100, note the memory budget: this package is ~5.3 GB of .hbm against that board's 12 GB of RAM (the S100P has 24 GB). It should fit alongside a normal ION carveout, but that has not been tried on an S100 — if you want the low-risk option there, the 0.8B build is ~2.2 GB.

cache_len is a performance knob as well as a capacity one — a larger KV window costs more DDR traffic per decoded token. Pick ctx512 if your prompts/replies genuinely fit in ~512 tokens; it decodes faster than ctx4k.

How to run

conda install -c https://mirrors.ruis.ai/conda -c conda-forge bllm

import bllm
m = bllm.load("s100p/ctx4k")
print(m.chat("这张图里有什么?", images=["photo.jpg"]))

# text-only, without loading the vision tower into memory
m = bllm.load("s100p/ctx4k", vision=False)
print(m.chat("介绍一下你自己"))

Requires a D-Robotics RDK S100 or S100P (matching folder, see above), sw >= 4.0.5, and an enlarged ION carveout (hb_switch_ion.sh balanced + reboot) — see BLLM's docs/LLM_ONBOARD.md.

Update 2026-11 — a short prompt reaches its first token 6.0–6.5x sooner

model_prefill.hbm and model.json were replaced in every folder; every other file is unchanged (the previous files are at revision 2c732b4d2e10).

A prompt is prefilled 32 tokens at a time. The part that did not fill a whole chunk — all of a typical question — used to go through the decode graph one token per step, and every step reads all of the model's weights from memory once, while one prefill chunk reads them once for 32 tokens. So time to first token was a cliff rather than a slope: a 31-token prompt took several times longer than a 32-token one. The new prefill graph takes that tail as one padded chunk: the pad rows leave the recurrent state untouched and the runtime takes their K/V back out of the attention window. Tails shorter than prefill_min_tail tokens (measured per folder, where stepping is still faster) are stepped as before.

folder	31-token prompt, before → after	prefill_min_tail	whole 32-token chunk (graph)	next-token top-1, padded vs stepped	first reply token, 12 chats
s100p/ctx4k	2.37 s → 0.36 s	5	331 → 337 ms	64/64	12/12
s100p/ctx512	2.05 s → 0.34 s	6	301 → 308 ms	63/64	12/12
s100/ctx4k	2.36 s → 0.36 s	5	331 → 336 ms	64/64	12/12
s100/ctx512	2.05 s → 0.34 s	6	302 → 308 ms	63/64	12/12

Measured on an RDK S100P (the s100/ builds run there, as in the tables below), feeding a text prompt up to the first token's logits. Wherever a chunk is full, the new graph gives bit-identical caches and logits to the previous one (every cache compared on its real values after 32 and 64 tokens). A padded tail and a stepped one are the same computation on two different int8 graphs (prefill vs decode), so they agree up to those graphs' numerics: next-token perplexity over 64 prompts whose tails cover every length (each target's log-probability read from the top 1000) moves by -0.28% to -0.18%, and prefill↔decode parity still passes in every folder.

Which bllm. The new graph has the same inputs and outputs as the old one, so an older bllm (≤ 0.7.0) runs these files exactly as it ran the previous ones — same speed, same answers. bllm 0.8.0 reads "prefill_pads": true from model.json and pads the tail; model.json also names the chat template family ("dialect": "qwen3.5"), which 0.8.0's OpenAI server uses for tool calls and reasoning_content.

To update: conda install -c https://mirrors.ruis.ai/conda -c conda-forge bllm=0.8.0, and hf download your folder again — only these two files changed.

Measured on hardware

s100p/ — RDK S100P, performance mode, 3 repeated runs

variant	decode	TTFT (image, coco-cats test photo)	prefill↔decode parity	ppl	image contrast	task anchors
ctx4096	13.20 tok/s	1.65 s	PASS	51.23	PASS	PASS
ctx512	15.29 tok/s	1.51 s	PASS	50.87	PASS	PASS

s100/ — the nash-e build, and how far it is verified

Compiled from the identical graph, weights and quantisation as the S100P build; only the BPU code generation differs. It was accepted by running it on S100P silicon (a nash-e binary loads and runs there) against the S100P build as a same-session paired baseline — same board, same harness, same prompts, greedy decoding:

build	decode	TTFT (image)	parity	ppl	image answer	contrast	anchors
s100/ctx4k (nash-e)	13.22 tok/s	1.866 s	PASS	51.2327	correct	PASS	PASS
s100p/ctx4k (nash-m, baseline)	13.21 tok/s	1.864 s	PASS	51.2327	correct	PASS	PASS
s100/ctx512 (nash-e)	15.30 tok/s	1.696 s	PASS	50.8687	correct	PASS	PASS
s100p/ctx512 (nash-m, baseline)	15.29 tok/s	1.708 s	PASS	50.8687	correct	PASS	PASS

Every generated answer was token-for-token identical between the two builds on every prompt, and the perplexities match to four decimals — the strongest statement available that the nash-e code generation did not change the model.

What is not verified: no RDK S100 board was available, so these are S100P timings, not S100 timings. An S100 is 80 TOPS with a 1.5 GHz CPU (vs 128 TOPS / 2.0 GHz), so expect it to be slower — by how much is unmeasured, and this card will not guess. See the memory note under "Which folder do I need?" as well.

Measured against this release's GQA batched-matmul + mask-dedup attention rewrite (replacing an expand_win broadcast) — essentially unchanged from the pre-rewrite baseline (13.22 tok/s), consistent with board output verified bit-identical to that baseline. Full methodology, acceptance protocol (prefill↔decode parity + perplexity + task-coherence anchors + with/without-image contrast — not just cosine similarity, see why in the model-zoo README), and every rejected build are in bllm-model-zoo/models/qwen3.5-2b and .../qwen3.5-vlm.

Known limitations

• The all-BPU chunked-prefill path (image ingestion, no CPU fallback) is numerically solvable but impractically slow to compile — this build's prefill graph uses a float [C,C] CPU fallback for one small intermediate instead. A compile-time/build-engineering tradeoff, not an accuracy or runtime limitation of the shipped model.
• int16 activation quantisation is data-free (no calibration set) at this size — verified sufficient at 2B; the 4B sibling needs per-linear calibration (see its model card).

License

The compiled .hbm is a derivative of the upstream Qwen3.5-2B weights and inherits its Apache-2.0 license — check the upstream model card for the current terms. BLLM (the runtime that loads this file) is open source; see its repository for its own license.