Jev-style decision model: GGUF BF16 / Q8_0 / Q4_K_M, client script, model card

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Jev-Style-Qwen3.5-2B-Decision-BF16.gguf filter=lfs diff=lfs merge=lfs -text
+Jev-Style-Qwen3.5-2B-Decision-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Jev-Style-Qwen3.5-2B-Decision-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+calibration.png filter=lfs diff=lfs merge=lfs -text

Jev-Style-Qwen3.5-2B-Decision-BF16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ccc6f44df660114180bf12a38ecef09cec9afe138f7776f2fb047aea5c2ce470
+size 3897379936

Jev-Style-Qwen3.5-2B-Decision-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3c14cd9d6d314d22f22ec527914e1cb7b5a9043823d4e815c0d232a9cd16cf6
+size 1312156768

Jev-Style-Qwen3.5-2B-Decision-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:470aa63b87fedf852e292580424fc4511778dffdbd459858f68abf4892dba040
+size 2076666976

README.md

@@ -0,0 +1,123 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.5-2B-Base
+pipeline_tag: text-generation
+language:
+- en
+tags:
+- gguf
+- jev-style
+- system-one
+- decision-model
+- calibration
+- classification
+- qwen3.5
+datasets:
+- nyu-mll/glue
+- fancyzhx/ag_news
+- google/boolq
+- SetFit/sst5
+---
+
+# Jev-Style-Qwen3.5-2B-Decision (GGUF)
+
+A **Jev-style decision model**: it does not write text. Give it a state, a question and a list of options, and it returns the decision **with calibrated probabilities** from a single token position. Runs in **LM Studio** and **llama.cpp**.
+
+| File | Size | Same decision as bf16 | Accuracy (500 held-out) |
+|---|---|---|---|
+| `Jev-Style-Qwen3.5-2B-Decision-Q4_K_M.gguf` | 1.3 GB | 94.4% | 82.4% |
+| `Jev-Style-Qwen3.5-2B-Decision-Q8_0.gguf` | 2.1 GB | 99.4% | 81.6% |
+| `Jev-Style-Qwen3.5-2B-Decision-BF16.gguf` | 3.9 GB | 99.8% | 81.6% |
+
+MLX bf16 build for Apple Silicon: [chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-MLX-bf16](https://huggingface.co/chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-MLX-bf16)
+
+## Results
+
+Everything below is measured on data the model never trained on, with the probabilities **exactly as the released weights produce them** (no post-processing).
+
+| | Qwen3.5-2B-Base, zero-shot | **This model** |
+|---|---|---|
+| Accuracy, 5 decision tasks (1,500 held-out examples) | 65.9% | **82.3%** |
+| Calibration error (ECE) on those tasks | 0.065 | **0.017** |
+| Negative log-likelihood / Brier score | 0.786 / 0.446 | **0.418 / 0.242** |
+| Calibration error on task types never seen in training | 0.155 | **0.075** |
+| Latency per decision (M1 Max, MLX bf16) | 76 ms | **77 ms** (no added cost) |
+
+- **Calibrated out of the box.** An ECE of 0.017 on 1,500 examples is statistically indistinguishable from a *perfectly* calibrated model: simulating labels from the model's own probabilities gives an expected ECE of 0.017 (95th percentile 0.025) from sampling noise alone. When this model says 80%, it is right about 80% of the time.
+- **Large accuracy gains where the base model struggled:** MNLI 52.3% -> 86.7%, SST-5 32.0% -> 61.7%, BoolQ 73.0% -> 82.7%, SST-2 87.3% -> 92.7%, AG News 84.7% -> 87.7%.
+- **Calibration transfers to new task types:** on emotion classification and RTE (never seen in training) the calibration error is halved (0.155 -> 0.075) at unchanged accuracy (64.5%).
+- **Zero-cost calibration.** A temperature fitted on 4,366 held-out examples is folded into the final RMSNorm weight, so every logit is already calibrated. Nothing to apply at inference time.
+- **Quantisation-friendly.** Q8_0 makes the same decision as bf16 on 99.4% of examples; Q4_K_M (1.3 GB) keeps 82.4% accuracy.
+- **Trained on a laptop.** The whole run (17.5k examples, LoRA rank 16 on all linear layers, log-score loss) took about 80 minutes on one M1 Max, using a custom chunk-parallel, differentiable Gated DeltaNet forward that matches the per-token training path to 1e-6 (outputs, state and all gradients) and is 6.5x faster per step (measured on the 0.8B sibling model).
+
+![Reliability diagram](calibration.png)
+## What "Jev-style" means
+
+[Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) (TypeSafe AI, 2026) introduced *System One* models: instead of generating text, the model takes a state plus a typed question and returns a **decision with calibrated probabilities** in a single pass. This model follows that pattern on top of an open base model:
+
+- **Choice** - pick one of N declared options, with a probability for each
+- **Bool** - probability that a proposition is true
+- **Score** - a distribution over ordered levels, and its expectation as a continuous score
+
+It cannot answer outside the declared options, it does not decode text, and one prefill pass gives the whole distribution.
+
+> This is an independent, from-scratch reproduction of the publicly described idea. It is not affiliated with TypeSafe AI and is not the Jev model.
+
+## Quick start
+
+**LM Studio** - download a file from this repo, load it, start the local server (Developer tab), then:
+
+```bash
+python jev_style_client.py --url http://localhost:1234 --model jev-style-qwen3.5-2b-decision
+```
+
+**llama.cpp**
+
+```bash
+llama-server -hf chaoliangUNSW/Jev-Style-Qwen3.5-2B-Decision-GGUF:Q8_0 --port 8080
+python jev_style_client.py --url http://localhost:8080
+```
+
+`jev_style_client.py` (standard library only) asks for one token with `top_logprobs` on `/v1/chat/completions` and renormalises the option letters:
+
+```python
+from jev_style_client import decide, decide_bool, decide_score
+
+decide("http://localhost:1234",
+       "Shares of the chipmaker jumped 8% after it raised its revenue forecast.",
+       "Which news section does this article belong to?",
+       ["World", "Sports", "Business", "Science/Technology"],
+       model="jev-style-qwen3.5-2b-decision")
+# [('Business', 0.68), ('Science/Technology', 0.31), ('World', 0.005), ('Sports', 0.002)]
+```
+
+Verified end to end through LM Studio's server on 500 held-out examples: 81.6% accuracy, ECE 0.028, about 110 ms per decision over HTTP on an M1 Max. In the LM Studio chat window you can also paste the prompt below and the model replies with the option letter.
+
+## Prompt format
+
+```
+You are a decision function. Read the state, then answer the question by choosing exactly one option.
+
+[State]
+{state}
+
+[Question]
+{question}
+
+[Options]
+A. {option 1}
+B. {option 2}
+
+Answer:
+```
+
+The next token is the option letter (` A`, ` B`, ...). Its probability, renormalised over the declared letters, is the decision distribution. For **Score**, list the levels in order; for **Bool**, use `yes` / `no`. The repository ships a pass-through chat template, so chat endpoints and the LM Studio chat window pass this text to the model verbatim.
+## Scope
+
+- A decision function, not a chat model: send the prompt format above.
+- Trained on five English task families (sentiment, natural-language inference, topic, yes/no question answering, 5-level rating). On unseen task types it keeps the base model's accuracy with better, though not perfect, calibration.
+- Up to 26 options (20 when probabilities are read through a server's `top_logprobs`).
+
+## Training data and licence
+
+SST-2 and MNLI (GLUE), AG News, BoolQ and SST-5, 22k examples converted to typed decisions; 80% for LoRA training, 20% held out for the calibration temperature. AG News is distributed for research / non-commercial use. Weights: Apache-2.0, same as [Qwen/Qwen3.5-2B-Base](https://huggingface.co/Qwen/Qwen3.5-2B-Base).

jev_style_client.py

@@ -0,0 +1,91 @@
+"""Typed decisions with calibrated probabilities from a local OpenAI-compatible server.
+
+The model never writes text: one token is requested, and the log-probs of the option
+letters at that position are renormalised into the decision distribution. The
+calibration temperature is already baked into the weights, so no post-processing
+beyond the renormalisation is needed.
+
+    # LM Studio:  load the model, start the local server, then
+    python jev_style_client.py --url http://localhost:1234 --model jev-style-qwen3.5-2b-decision
+    # llama.cpp:  llama-server -m Jev-Style-Qwen3.5-2B-Decision-Q8_0.gguf --port 8080
+    python jev_style_client.py --url http://localhost:8080
+
+Only the Python standard library is used.
+"""
+
+import argparse
+import json
+import math
+import urllib.request
+
+LETTERS = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+HEADER = ("You are a decision function. Read the state, then answer the question "
+          "by choosing exactly one option.\n\n")
+MAX_OPTIONS = 20  # servers return at most ~20 candidate tokens per position
+
+
+def build_prompt(state, question, options):
+    lines = "\n".join(f"{LETTERS[i]}. {o}" for i, o in enumerate(options))
+    return f"{HEADER}[State]\n{state}\n\n[Question]\n{question}\n\n[Options]\n{lines}\n\nAnswer:"
+
+
+def _post(url, payload):
+    req = urllib.request.Request(url, json.dumps(payload).encode(), {"Content-Type": "application/json"})
+    with urllib.request.urlopen(req, timeout=120) as r:
+        return json.loads(r.read())
+
+
+def _top_logprobs(base_url, prompt, model):
+    """-> {token_text: logprob} for the first generated position.
+
+    Uses /v1/chat/completions: the released model ships a pass-through chat template, so the
+    prompt reaches the model verbatim, and both LM Studio and llama-server return candidate
+    log-probs on this endpoint (LM Studio does not on /v1/completions).
+    """
+    out = _post(base_url.rstrip("/") + "/v1/chat/completions",
+                {"model": model, "messages": [{"role": "user", "content": prompt}], "max_tokens": 1,
+                 "temperature": 0, "logprobs": True, "top_logprobs": 20})
+    return {t["token"]: t["logprob"] for t in out["choices"][0]["logprobs"]["content"][0]["top_logprobs"]}
+
+
+def decide(base_url, state, question, options, model="jev-style"):
+    """Choice: -> list of (option, probability), most likely first."""
+    assert 2 <= len(options) <= MAX_OPTIONS
+    top = _top_logprobs(base_url, build_prompt(state, question, options), model)
+    floor = min(top.values()) - 5.0  # option letter outside the returned candidates: negligible mass
+    logits = [max((lp for tok, lp in top.items() if tok.strip() == LETTERS[i]), default=floor)
+              for i in range(len(options))]
+    z = max(logits)
+    exp = [math.exp(x - z) for x in logits]
+    probs = [e / sum(exp) for e in exp]
+    return sorted(zip(options, probs), key=lambda t: -t[1])
+
+
+def decide_bool(base_url, state, proposition, model="jev-style"):
+    """Bool: -> probability that the proposition is true."""
+    return dict(decide(base_url, state, proposition, ["yes", "no"], model))["yes"]
+
+
+def decide_score(base_url, state, question, levels, model="jev-style"):
+    """Score: ordered levels -> (expected level index in [0, len-1], distribution)."""
+    dist = dict(decide(base_url, state, question, levels, model))
+    return sum(i * dist[l] for i, l in enumerate(levels)), [(l, dist[l]) for l in levels]
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--url", default="http://localhost:1234")
+    ap.add_argument("--model", default="jev-style-qwen3.5-2b-decision")
+    args = ap.parse_args()
+    url, mid = args.url, args.model
+
+    review = "The plot is thin, but the two leads are so charming that I left the cinema smiling."
+    print("Choice:", decide(url, review, "What is the sentiment of this review?", ["negative", "positive"], mid))
+    print("Score :", decide_score(url, review, "Rate the sentiment of this review on an ordered scale.",
+                                  ["very negative", "negative", "neutral", "positive", "very positive"], mid))
+    print("Bool  :", decide_bool(
+        url, "Premise: A man is playing a guitar on stage.\nHypothesis: Someone is performing music.",
+        "Does the premise entail the hypothesis?", mid))
+    news = "Shares of the chipmaker jumped 8% after it raised its full-year revenue forecast."
+    print("Choice:", decide(url, news, "Which news section does this article belong to?",
+                            ["World", "Sports", "Business", "Science/Technology"], mid))