What is this?
This is an abliterated (heretic) DELLA merge of the following finetuned & post-trained models:
| Model Repo | Ratio | Purpose |
|---|---|---|
| Qwen/Qwen3.6-35B-A3B | (base) | The base model, used for calculating task vectors. |
| Kwaipilot/KAT-Coder-V2.5-Dev | 0.58 | Contributes strong token-efficient coding & bugfixing |
| InternScience/Sage-A1.5 | 0.30 | Contributes strong instruction following (Sage has 81.6 on IFBench!), as well as agent orchestration, bugfixing, verification, planning |
| ornith-ai/Ornith-1.5-35B-A3B | 0.12 | Contributes self-scaffolding, planning, coding |
The result is an abliterated hybrid model for coding, bugfixing, agent work, planning, designing, conversation, data-driven RP, game logic/procgen. The model plans out its steps before acting, and it verifies & fixes its work after it drafts a solution.
V1.1: Compared to the first version, V1.1 was created with adjusted ratios and parameters. Sage seems to overpower KAT-Coder a bit too much so I biased the merge further towards KAT-Coder, but I also raised Ornith's ratio slightly.
I also raised density slightly to 0.7, and epsilon raised to 0.1, except for protected routing and attention tensors, which remain at 1.0 & 0.0 respectively. Finally, I raised lambda to 1.05 to slightly boost the task vectors and counteract "flattening".
The model should overthink less, make fewer mistakes, and show somewhat lower perplexity (llama-perplexity reported an average of 2.3539 +/- 0.03358 PPL running the BF16 model on my evaluation set of held-out wikitext, code, documents, and reasoning traces).
In a proper environment, this model can be extremely persistent in achieving the task given to it; ensure proper sandboxing before agentic use.
Quantization
Weights are provided quantized with APEX, which generates mixed-precision quantization configurations for MoE models.
The imatrix is calibrated mostly on a mixture of English wikitext, documents, code, and tokens generated during real long-context agentic coding sessions with KAT-Coder, Ornith-1.5, and the model itself (control tokens such as <|im_start|> included).
The configs used to quantize these models were modified by hand based on my own research as well as trial-and-error, with care taken to protect the outer layers and sensitive tensors, even if APEX doesn't normally protect them.
I-Quality (~23 GB)
This is a custom configuration based on I-Quality but with sensitive tensors bumped up to BF16/Q8_0/Q6_K and the routed experts in middle layers stored at IQ3_S through Q6_K. Optimized for partial CPU offload and high quality long-context work.
KL divergence & same-top% vs BF16 baseline, as measured by llama-perplexity on a mixture of wikitext, code, markdown & xml documents, and reasoning traces, all held out from calibration:
====== KL divergence statistics ======
Mean KLD: 0.027424 ± 0.001668
Maximum KLD: 14.057955
99.9% KLD: 3.334642
99.0% KLD: 0.321866
95.0% KLD: 0.079816
90.0% KLD: 0.040109
Median KLD: 0.000707
10.0% KLD: -0.000372
5.0% KLD: -0.000702
1.0% KLD: -0.001460
0.1% KLD: -0.002148
Minimum KLD: -0.002832
====== Token probability statistics ======
Mean Δp: -0.010 ± 0.031 %
Maximum Δp: 99.278%
99.9% Δp: 44.593%
99.0% Δp: 12.560%
95.0% Δp: 3.764%
90.0% Δp: 1.059%
75.0% Δp: 0.112%
Median Δp: -0.003%
25.0% Δp: -0.146%
10.0% Δp: -1.202%
5.0% Δp: -3.899%
1.0% Δp: -13.006%
0.1% Δp: -34.061%
Minimum Δp: -99.898%
RMS Δp : 4.665 ± 0.189 %
Same top p: 96.181 ± 0.128 %
I-Medium (~20 GB)
This is a custom configuration that is somewhere between I-Compact and I-Quality. Sensitive tensors bumped up to BF16/Q8_0/Q6_K, and routed experts are stored at IQ3_XXS through IQ4_NL in the middle layers. Optimized for partial CPU offload, and medium quality long-context work.
KL divergence & same-top% vs BF16 baseline, as measured by llama-perplexity on a mixture of wikitext, code, markdown & xml documents, and reasoning traces, all held out from calibration:
====== KL divergence statistics ======
Mean KLD: 0.038313 ± 0.002027
Maximum KLD: 13.475267
99.9% KLD: 4.493835
99.0% KLD: 0.492805
95.0% KLD: 0.117759
90.0% KLD: 0.058738
Median KLD: 0.000800
10.0% KLD: -0.000450
5.0% KLD: -0.000841
1.0% KLD: -0.001648
0.1% KLD: -0.002258
Minimum KLD: -0.002751
====== Token probability statistics ======
Mean Δp: -0.069 ± 0.038 %
Maximum Δp: 99.137%
99.9% Δp: 55.203%
99.0% Δp: 15.870%
95.0% Δp: 4.240%
90.0% Δp: 1.181%
75.0% Δp: 0.134%
Median Δp: -0.001%
25.0% Δp: -0.162%
10.0% Δp: -1.527%
5.0% Δp: -4.793%
1.0% Δp: -17.197%
0.1% Δp: -49.123%
Minimum Δp: -99.385%
RMS Δp : 5.655 ± 0.187 %
Same top p: 95.443 ± 0.139 %
(More coming soon!)
- Downloads last month
- 1,311