CMSManhattan/JiRackTernaryPrecision-36B-chat

JiRack Ternary Precision 36B (GGUF)

JiRack Ternary Precision 36B is a compact ternary (1.58-bit) model for agents, function calling and coding by CMSManhattan (Konstantin Grabko), with 35.78B parameters. Every linear layer of the transformer uses ternary weights {-1, 0, +1} with a per-tensor scale and 8-bit activations (BitNet b1.58 style), trained with quantization-aware training.

• Function calling: BFCL v3 non-live 80.4% (1,390 cases) and live 56.1% (2,251 cases) with the Q2_0 GGUF file; 109/117 (93.2%) on our 117-case tool-calling test.
• Coding: HumanEval 48.2% (Python) and MultiPL-E 47.1% over Java, JavaScript, TypeScript, C++, Go and Dart.
• Math: GSM8K 76.0% (first 300 test problems).
• Small and fast: on a GPU the model needs about 11 GiB of memory and generates 108.5 tokens/s for one user. It also runs on a plain CPU.
• GGUF files with the same weights: TQ2_0 (9.91 GB) for CPUs, Q2_0 (10.71 GB) for NVIDIA GPUs, TQ1_0 (8.31 GB) with denser packing. All of them store the ternary weights exactly.

Live demo (CPU, TQ2_0): JiRack-36B-Chat

Which NVIDIA Jetson can hold each model (estimate)

Estimated from file sizes plus the KV cache for a 4k context, with about 2 GB of the shared memory left to the operating system. Not yet measured on Jetson devices.

Model	GGUF TQ2_0	Fits	Tight (short context)	Does not fit
2B	0.73 GB	Orin Nano 4/8 GB, Orin NX, AGX Orin, Thor	—	—
8B	2.6 GB	Orin Nano 8 GB, Orin NX 8/16 GB, AGX Orin, Thor	—	Orin Nano 4 GB
16B	4.8 GB	Orin NX 16 GB, AGX Orin, Thor	Orin Nano 8 GB, Orin NX 8 GB	Orin Nano 4 GB
36B	9.91 GB	AGX Orin 32/64 GB, Thor	Orin NX 16 GB	Orin Nano, Orin NX 8 GB

On Jetson, llama.cpp runs TQ2_0 on the CPU cores. To run on the Jetson GPU (CUDA), use the Q2_0 file of the same model: the same ternary weights stored exactly, slightly larger (36B: 10.71 GB).

Results

Internal tool-calling test (117 cases, greedy)

The cases are written by us and are not in the training data. The GGUF file is served by llama-server through jirack_gguf_proxy.py, with the same prompt format as in fine-tuning.

Category	Reference (PyTorch, ternary)	GGUF Q2_0 (GPU)
Single call (one tool, all arguments given)	38/38	38/38
Parallel calls	19/19	18/19
Follow-up (answer from a tool result)	20/20	20/20
No tool needed	25/25	25/25
Clarify a missing argument	14/15	8/15
Total	116/117 (99.1%)	109/117 (93.2%)

The GGUF file holds exactly the same ternary weights. llama.cpp computes the 8-bit activations slightly differently from the reference code, and this flips some borderline decisions. All 8 differences are of this kind: in 7 of the 15 clarify cases the GGUF model called the tool instead of asking for the missing argument, and in one parallel case it made one call instead of two.

BFCL v3 (Berkeley Function Calling Leaderboard)

Category	Cases	JiRack Ternary 36B, Q2_0	DeepSeek-R1-Distill-Qwen-32B (bf16)
simple	400	91.5%	75.0%
multiple	200	93.0%	83.0%
parallel	200	85.5%	55.5%
parallel_multiple	200	73.0%	72.5%
java	100	62.0%	52.0%
javascript	50	56.0%	68.0%
irrelevance	240	66.3%	93.3%
Non-live overall	1,390	80.4%	74.2%
live_simple	258	71.7%	75.2%
live_multiple	1,053	62.6%	69.8%
live_parallel	16	43.8%	43.8%
live_parallel_multiple	24	50.0%	58.3%
live_relevance	18	100.0%	38.9%
live_irrelevance	882	43.2%	90.3%
Live overall	2,251	56.1%	77.9%

Overall = average weighted by the number of cases (evalscope 1.14.0 with bfcl-eval 2026.3.23). JiRack: the Q2_0 GGUF file on llama-server (llama.cpp c6a0ff11a, NVIDIA RTX PRO 6000 Blackwell) behind jirack_gguf_proxy.py, BFCL function-calling mode (tools passed as OpenAI tools), greedy, up to 1,024 new tokens. DeepSeek-R1-Distill-Qwen-32B: bf16 GGUF on llama-server, BFCL prompt mode, its own chat template with reasoning on, temperature 0.6 / top-p 0.95 as DeepSeek recommends, up to 4,096 new tokens. DeepSeek-R1-Distill-Qwen-32B is much better at declining when no tool fits (irrelevance, live_irrelevance); see Limitations.

Compared with other open models at 2 bits

All files on one NVIDIA RTX PRO 6000 Blackwell (96 GB) in Google Colab, the same llama.cpp build (c6a0ff11a), every layer on the GPU, the same tests. Speed: llama-bench, one user, 512-token prompt (pp512), 128 generated tokens (tg128).

Model, file	Size	Bits per weight	Generation, tokens/s	Prompt, tokens/s	Tool-calling test	BFCL v3 non-live	BFCL v3 live
JiRack Ternary 36B, Q2_0	9.97 GiB	2.4	107.9	4,062	109/117	80.4%	56.1%
Gemma 4 31B, Q2_K	11.08 GiB	3.1	88.9	2,883	115/117	87.4%	78.7%
Muse Glimmer 30B, Q2_K	9.94 GiB	3.1	108.8	3,289	97/117	55.7%¹	71.2%¹
Gemma 4 31B, Q8_0	30.38 GiB	8.5	41.3	3,975	113/117	87.6%	78.9%
Muse Glimmer 30B, Q8_0	27.57 GiB	8.5	48.3	5,094	96/117	58.1%¹	71.3%¹

• Our weights are ternary by training, so the 2-bit file is the model exactly as trained. The other two models are compressed after training: Q2_K with an importance matrix that we computed from their Q8_0 files (llama-imatrix). Q2_K keeps the most sensitive tensors at higher precision, so it averages about 3.1 bits per weight (file size divided by the number of parameters); our file averages about 2.4. The Q8_0 rows show what 2 bits cost them: almost nothing for Gemma 4 31B, 2.4 points of BFCL non-live for Muse Glimmer 30B.
• ¹ Muse Glimmer 30B makes one tool call per turn, so it scores close to zero on the parallel-call categories of BFCL (non-live parallel 3% and parallel_multiple 1%, live parallel 0%) and 0 (Q8_0) or 1 (Q2_K) of 19 in the parallel part of our test. Without the parallel categories, BFCL non-live is 86.4% for Gemma 4 31B, 80.9% for our model and 77.4% for Muse Glimmer 30B (2-bit files).
• Where we stand at 2 bits: our model reads prompts fastest, generates as fast as the fastest and uses the fewest bits per weight. Gemma 4 31B is more accurate on every quality test, most of all on BFCL live (78.7% against our 56.1%). Against Muse Glimmer 30B we lead on the tool-calling test and BFCL non-live, it leads on BFCL live.
• Settings of the other models: tool calls through llama.cpp's own function calling (llama-server --jinja, tools as OpenAI tools), greedy, up to 4,096 new tokens; BFCL v3 with the same evalscope and bfcl-eval versions as above.

Coding and math

Benchmark	Problems	Solved	Score
HumanEval (Python)	164	79	48.2%
MultiPL-E HumanEval, Java	158	71	44.9%
MultiPL-E HumanEval, JavaScript	161	81	50.3%
MultiPL-E HumanEval, TypeScript	159	89	56.0%
MultiPL-E HumanEval, C++	161	70	43.5%
MultiPL-E HumanEval, Go	154	62	40.3%
MultiPL-E HumanEval, Dart	157	74	47.1%
MultiPL-E, all six languages	950	447	47.1%
GSM8K (first 300 test problems)	300	228	76.0%

Reference model (the same ternary weights as the GGUF files, 8-bit activations), greedy decoding, pass@1 with the official tests. HumanEval and GSM8K: plain chat prompt without a system prompt, up to 1,024 new tokens; GSM8K asks for the final answer in \boxed{} and checks it as an exact number. MultiPL-E: HumanEval translated to six languages, each checked with its own tests, up to 768 new tokens.

Speed and memory on a GPU

NVIDIA GH200 (Hopper GPU), llama.cpp CUDA build ac8e1694d, all layers on the GPU. Peak GPU memory: nvidia-smi, sampled once per second during the run. Measured with the Q2_0 file of the previous release: the file format, size and architecture are the same, so speed and memory are too.

	JiRack Ternary 36B, Q2_0	DeepSeek-R1-Distill-Qwen-32B, BF16
File	9.97 GiB	61.03 GiB
Peak GPU memory, 1 stream (llama-bench)	10.7 GiB	60.7 GiB
Peak GPU memory, 32 parallel streams	20.5 GiB	70.7 GiB
Generation, 1 stream (tg128)	108.5 tokens/s	51.3 tokens/s
Generation, 8 parallel streams (total)	363 tokens/s	298 tokens/s
Generation, 32 parallel streams (total)	1,050 tokens/s	1,297 tokens/s
Prompt processing (pp512)	2,732 tokens/s	6,988 tokens/s

Parallel streams: llama-batched-bench, 512 prompt + 128 generated tokens per stream. The ternary model needs 5.7× less memory (it fits a 24 GB GPU even with 32 parallel streams) and answers a single user about 2× faster. With many parallel streams and for long prompts the bf16 model is faster on a Hopper GPU, because its matrix multiplications run on the tensor cores while Q2_0 unpacks the ternary weights first.

This release's Q2_0 file on an NVIDIA RTX PRO 6000 Blackwell (llama-bench, llama.cpp c6a0ff11a, all layers on the GPU): prompt processing (pp512) 4,062 tokens/s, generation (tg128) 107.9 tokens/s.

Speed on a CPU

llama-bench on the CPU of an NVIDIA GH200 node (Grace, 64 threads), CPU-only llama.cpp build, files of the previous release (same format and size):

File	Prompt processing (512 tokens)	Generation
TQ2_0	67.7 tokens/s	20.7 tokens/s
TQ1_0	45.7 tokens/s	24.5 tokens/s

Files

File	Size	Notes
JiRackTernaryPrecision-36B-TQ2_0.gguf	9.91 GB	recommended for CPUs
JiRackTernaryPrecision-36B-Q2_0.gguf	10.71 GB	recommended for NVIDIA GPUs (CUDA); same ternary weights, stored exactly; tested (above)
JiRackTernaryPrecision-36B-TQ1_0.gguf	8.31 GB	same ternary weights, denser packing; not separately evaluated
jirack_gguf_proxy.py + jirack_openai_server.py		OpenAI-compatible endpoint with tool calls on top of llama-server
chat_template.jinja		chat template embedded in the GGUF (plain chat); for tool calls use the proxy
eval/		tool-calling test and BFCL logs of the GGUF file, reference-model results, llama-bench
NOTICE.md		third-party licenses

Run with llama.cpp

Plain chat on a CPU:

llama-cli -m JiRackTernaryPrecision-36B-TQ2_0.gguf -c 8192 -cnv

On an NVIDIA GPU use the Q2_0 file and a recent llama.cpp built with CUDA (-DGGML_CUDA=ON, a build that knows the Q2_0 type), with all layers on the GPU:

llama-server -m JiRackTernaryPrecision-36B-Q2_0.gguf -ngl 99 --port 8080 -c 32768 --special

Tool calling through an OpenAI-compatible API. llama-server returns the model's tool calls as text (<|tool_call_start|> {"name": ..., "arguments": ...} <|tool_call_end|>); the proxy builds the exact fine-tuning prompt and turns the calls into standard OpenAI tool_calls, so any OpenAI client or agent framework can use the model:

llama-server -m JiRackTernaryPrecision-36B-TQ2_0.gguf --port 8080 -c 32768 --special   # GPU: Q2_0 file + -ngl 99
python3 jirack_gguf_proxy.py --llama http://127.0.0.1:8080 --port 8001   # needs jirack_openai_server.py next to it

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="none")
tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
r = client.chat.completions.create(model="jirack-36b", tools=tools,
                                   messages=[{"role": "user", "content": "Weather in Austin?"}])
print(r.choices[0].message.tool_calls)

Return tool results as tool messages; several calls in one turn (parallel calls) are supported. Use greedy decoding (temperature 0) for tool calls.

Model details

Parameters	35.78B
Layers	64
Hidden size / FFN	5,120 / 30,720
Attention heads	40 query, 8 key-value (GQA), head dim 128
Vocabulary	152,064
Context	up to 32k tokens
Weights	ternary {-1, 0, +1}, per-tensor scale
Activations	8-bit, per token

The GGUF files store every ternary weight exactly as the model computes it (clamp(round(W / mean|W|), -1, 1) * mean|W|, 448 matrices), packed as TQ2_0 / TQ1_0 (CPU) and Q2_0 (GPU).

How the model was made

JiRack Ternary Precision 36B is a modified version of a Qwen model by the Qwen team, Alibaba Cloud, licensed under Apache 2.0 (license text); its tokenizer and chat format come from a DeepSeek model by DeepSeek (MIT). CMSManhattan modified the architecture and the weights, trained every linear layer to ternary weights with 8-bit activations (quantization-aware training) and fine-tuned the model for coding, tool calling and agents.

Intended use

Tool calling, function calling, coding assistants and agent pipelines; local, offline and on-premise deployment where GPU memory or hardware cost is limited.

Limitations

• The model calls a tool more often than it should: when no available tool fits the request (BFCL irrelevance 66.3%, live_irrelevance 43.2% with the Q2_0 file) or when a required argument is missing (clarify 8/15 with the Q2_0 file, 14/15 with the reference model). Validate calls before executing them, and give the agent an explicit way to answer without a tool.
• Generated code passes the tests in about half of the HumanEval / MultiPL-E problems: review and test it before use.
• The model can make mistakes, invent facts and produce wrong tool arguments. Check its output before acting on it, especially in automated pipelines.
• Ternary model: world knowledge is below that of a bf16 model of the same size; not for medical, legal or financial advice.

ROBOS HUMANOID Project

• https://github.com/cmsmanhattan/ROBOS

📧 Contact

For joint venture opportunities, hardware integration, or licensing inquiries:

• Email: [email protected]
• Phone: +1 (516) 777-0945
• Location: New York, USA

License

© 2025-2026 Konstantin Vladimirovich Grabko, all rights reserved.