Release unmodified quantization with complete saved evaluation metrics and historical reference

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

EVALUATION.md

@@ -0,0 +1,88 @@
+# VeriLoop-E2 paired quantized evaluation
+
+Same frozen questions, seed 20260816, thinking enabled, publisher raw-content protocol, 32768-token output budget. Legacy custom scoring, including partial IFEval rules. Format comparisons use different engines/KV types; compare original versus ablated within each format.
+
+Runtime: dual RTX 3090. W4A16 uses vLLM 0.29.0 with a local FA2/FP8-KV plugin, tensor parallel 2, concurrency 16, 65,536-token context, FP8 E5M2 KV. GGUF uses llama.cpp layer split 1:1, concurrency 8, 40,960-token context per slot, Q8_0 key/value cache. Both use temperature 0 and top_p 1. See evaluation-runtime-versions.json for exact runtime identifiers.
+
+## w4a16
+
+| Task | n | Original | Ablated | Delta (pp) |
+|---|---:|---:|---:|---:|
+| mmlu | 150 | 84.00% | 76.00% | -8.00 |
+| cmmlu | 150 | 84.67% | 75.33% | -9.33 |
+| mmlu_pro | 150 | 78.67% | 70.67% | -8.00 |
+| ceval | 150 | 84.00% | 76.00% | -8.00 |
+| arc | 150 | 80.67% | 85.33% | +4.67 |
+| truthfulqa | 150 | 58.67% | 71.33% | +12.67 |
+| gsm8k | 100 | 95.00% | 88.00% | -7.00 |
+| math500 | 100 | 75.00% | 73.00% | -2.00 |
+| bbh | 150 | 75.33% | 65.33% | -10.00 |
+| humaneval | 164 | 89.63% | 89.63% | +0.00 |
+| ifeval | 100 | 76.00% | 69.00% | -7.00 |
+
+Overall: 1212/1514 (80.05%) → 1157/1514 (76.42%). Delta -3.63 pp, paired approximate 95% CI [-5.894, -1.371] pp. Lost/gained: 181/126.
+Empty: [9, 8]; explicitly delimited reasoning: [1186, 1007]; length-limited: [9, 8]. Empty-rate validity gate (≤2%): False.
+Refusal head markers (100 prompts, thinking off): [98, 0]; validity: [True, True]; 2048-token budget hits: [1, 9]. See comparison.json for all marker counts.
+
+Empty-rate gate exception: original-w4a16 / humaneval: 5/164 (3.05%), above 2%. These outputs remain failed in the denominator.
+### Refusal marker diagnostics
+
+| Metric | Original | Ablated |
+|---|---:|---:|
+| substring_any | 98 | 2 |
+| substring_head | 98 | 0 |
+| heretic33_any | 98 | 69 |
+| ega26_any | 98 | 55 |
+| short | 0 | 0 |
+| empty | 0 | 0 |
+| empty_but_reasoning | 0 | 0 |
+| reasoning_leaked | 0 | 0 |
+| errors | 0 | 0 |
+
+Selection-independent 84-item subset: opening markers [82, 0]; anywhere markers [82, 2]. These are heuristic counts, not a semantic compliance score.
+
+## gguf
+
+| Task | n | Original | Ablated | Delta (pp) |
+|---|---:|---:|---:|---:|
+| mmlu | 150 | 68.00% | 68.00% | +0.00 |
+| cmmlu | 150 | 61.33% | 64.67% | +3.33 |
+| mmlu_pro | 150 | 64.00% | 68.67% | +4.67 |
+| ceval | 150 | 60.00% | 70.00% | +10.00 |
+| arc | 150 | 90.00% | 91.33% | +1.33 |
+| truthfulqa | 150 | 60.67% | 60.67% | +0.00 |
+| gsm8k | 100 | 37.00% | 47.00% | +10.00 |
+| math500 | 100 | 68.00% | 67.00% | -1.00 |
+| bbh | 150 | 48.00% | 54.67% | +6.67 |
+| humaneval | 164 | 90.85% | 90.24% | -0.61 |
+| ifeval | 100 | 63.00% | 60.00% | -3.00 |
+
+Overall: 995/1514 (65.72%) → 1039/1514 (68.63%). Delta +2.91 pp, paired approximate 95% CI [0.956, 4.856] pp. Lost/gained: 92/136.
+Empty: [4, 7]; explicitly delimited reasoning: [741, 782]; length-limited: [4, 7]. Empty-rate validity gate (≤2%): True.
+Refusal head markers (100 prompts, thinking off): [99, 0]; validity: [True, True]; 2048-token budget hits: [0, 7]. See comparison.json for all marker counts.
+
+### Refusal marker diagnostics
+
+| Metric | Original | Ablated |
+|---|---:|---:|
+| substring_any | 99 | 0 |
+| substring_head | 99 | 0 |
+| heretic33_any | 99 | 69 |
+| ega26_any | 99 | 54 |
+| short | 0 | 0 |
+| empty | 0 | 0 |
+| empty_but_reasoning | 0 | 0 |
+| reasoning_leaked | 0 | 0 |
+| errors | 0 | 0 |
+
+Selection-independent 84-item subset: opening markers [83, 0]; anywhere markers [83, 0]. These are heuristic counts, not a semantic compliance score.
+
+## Limits
+
+An initial original-W4A16 preflight incorrectly enabled a Qwen3 reasoning parser that the publisher does not prescribe. It classified 33/150 direct answers as hidden reasoning. Removing the incompatible parser recovered all 33, with zero empty/length-limited outputs; streaming and nonstreaming probes passed. All reported runs keep thinking enabled and the original template, use publisher raw-content semantics, and extract final content after a closing think tag only when present. Unclosed length-limited output is not scored as an answer.
+
+These are local sampled benchmarks with the historical custom scorers, not official leaderboard scores. Truncated or empty outputs remain in the primary denominator. Marker absence does not prove semantic compliance. Refusal probes overlap the earlier 16-item selection set for the first 16 items; the remaining 84 are not used for candidate selection. No new ablation tuning was performed after these evaluations. No equivalence claim follows solely from a nonsignificant difference.
+
+GGUF protocol verification: the supplied standalone source template overrides an older embedded identity block. All 1,514 prompts and token sequences match the source tokenizer. llama.cpp echoes the opening think generation-prefix tag in content-only mode; the protocol adapter removes that framing before legacy scoring. Two streaming/nonstreaming probes passed. The partial GGUF first pass is excluded. Requests allow 7,200 seconds for the slower engine, while output budgets and scorers remain unchanged.
+
+The fixed-protocol GGUF run was paused to investigate low GSM8K accuracy. Two failed cases remained wrong with prompt caching disabled and with CPU-only F16 KV. One also failed under the original Hugging Face BF16 model in Transformers. Sampled BF16 GGUF tensors matched the source exactly. These checks did not establish a GGUF-only pipeline fault; they do not establish full BF16 benchmark accuracy or explain the entire cross-format gap. The existing run resumed without changing questions, scoring, weights, or engine. Wall-clock durations include the diagnostic pause and are not throughput measurements. An isolated newer llama.cpp CPU build was not used for the reported evaluation.

HISTORICAL-COMPARISON.md

@@ -0,0 +1,48 @@
+# Measured results and historical reference
+
+Columns Qwen base and Qwen community refer to our historical W4A16 quantizations, **not BF16 originals**. Qwen community uses JonathanColetti-Qwen3.8-27B-Uncensored upstream weights; it is not an abliteration developed by this publisher. VeriLoop base means the unmodified upstream model quantized by us. Ablated columns are evaluation records only.
+
+| Task | Qwen base W4A16 | Qwen community W4A16 | VeriLoop base W4A16 | VeriLoop ablated W4A16 | VeriLoop base GGUF | VeriLoop ablated GGUF |
+|---|---:|---:|---:|---:|---:|---:|
+| mmlu | 130/150 (86.67%) | 126/150 (84.00%) | 126/150 (84.00%) | 114/150 (76.00%) | 102/150 (68.00%) | 102/150 (68.00%) |
+| cmmlu | 135/150 (90.00%) | 134/150 (89.33%) | 127/150 (84.67%) | 113/150 (75.33%) | 92/150 (61.33%) | 97/150 (64.67%) |
+| mmlu_pro | 132/150 (88.00%) | 125/150 (83.33%) | 118/150 (78.67%) | 106/150 (70.67%) | 96/150 (64.00%) | 103/150 (68.67%) |
+| ceval | 127/150 (84.67%) | 127/150 (84.67%) | 126/150 (84.00%) | 114/150 (76.00%) | 90/150 (60.00%) | 105/150 (70.00%) |
+| arc | 144/150 (96.00%) | 144/150 (96.00%) | 121/150 (80.67%) | 128/150 (85.33%) | 135/150 (90.00%) | 137/150 (91.33%) |
+| truthfulqa | 132/150 (88.00%) | 120/150 (80.00%) | 88/150 (58.67%) | 107/150 (71.33%) | 91/150 (60.67%) | 91/150 (60.67%) |
+| gsm8k | 99/100 (99.00%) | 99/100 (99.00%) | 95/100 (95.00%) | 88/100 (88.00%) | 37/100 (37.00%) | 47/100 (47.00%) |
+| math500 | 75/100 (75.00%) | 77/100 (77.00%) | 75/100 (75.00%) | 73/100 (73.00%) | 68/100 (68.00%) | 67/100 (67.00%) |
+| bbh | 140/150 (93.33%) | 138/150 (92.00%) | 113/150 (75.33%) | 98/150 (65.33%) | 72/150 (48.00%) | 82/150 (54.67%) |
+| humaneval | 143/164 (87.20%) | 137/164 (83.54%) | 147/164 (89.63%) | 147/164 (89.63%) | 149/164 (90.85%) | 148/164 (90.24%) |
+| ifeval | 89/100 (89.00%) | 93/100 (93.00%) | 76/100 (76.00%) | 69/100 (69.00%) | 63/100 (63.00%) | 60/100 (60.00%) |
+| **Total** | **1346/1514 (88.90%)** | **1320/1514 (87.19%)** | **1212/1514 (80.05%)** | **1157/1514 (76.42%)** | **995/1514 (65.72%)** | **1039/1514 (68.63%)** |
+
+## Comparison boundary
+
+Historical runs were on 2026-08-30; VeriLoop runs were on 2026-10-10. Both use seed 20260816, the same 11 categories and 1,514 samples, thinking enabled, greedy decoding and a 32,768-token output budget. `evaluation-item-alignment.json` records exact ordered item-ID comparisons against the current frozen set. Matching IDs do not establish identical dataset revisions or prompt bytes in the historical run.
+
+Historical W4A16 used a 262,144-token context, a previous vLLM environment and the Qwen3 reasoning parser. Current W4A16 used context 65,536, vLLM 0.29.0, and raw-content extraction without that parser. Both used tensor parallelism 2, concurrency 16 and FP8 E5M2 KV cache. Current GGUF used llama.cpp, layer split 1:1, concurrency 8, context 40,960 per slot and Q8_0 KV cache. These are historical reference comparisons, **not strictly identical-runtime controlled experiments**. The GGUF/W4A16 difference also includes engine and KV-cache differences.
+
+There is no full BF16 baseline for VeriLoop in this evaluation. These sampled/custom scores are not official leaderboard scores and do not reproduce the upstream private VeriLoop Harness or its agent/tool benchmarks. No conclusion about the validity of the authors' reported results follows from this table.
+
+## Data coverage
+
+`evaluation-items.json` contains every saved per-item metric for all six runs (9,084 records), including IDs, correctness, token counts, errors, and available truncation/finish metadata. Saved answer tails and benchmark gold text are excluded. The original harness retained only answer tails, not complete generated responses; complete response replay is therefore unavailable. `evaluation-question-indices.json` supplies all current IDs and prompt hashes, without redistributing benchmark question text. Dataset loaders and scoring rules are supplied in `evaluation-legacy-scorer.py` (the frozen implementation, including historical comments), with `evaluation-protocol.py` and `evaluation-adapter.py` documenting current extraction and code execution. Dataset rights remain with their owners; model Apache-2.0 licensing does not relicense third-party datasets.
+
+`evaluation-refusal-items.json` contains the saved metadata for 400 refusal probes, with no harmful prompts or generated answers. Marker counts are heuristic measurements, not semantic compliance judgments. No ablation implementation or weight-modification recipe is included in this release.
+
+## Standards and validity
+
+Each category score is correct/n; total is sum(correct)/1514, not an unweighted average of category percentages. Main scores retain empty and truncated cases in the denominator. The historical summaries did not record the newer empty and finish-reason fields; absent fields must not be interpreted as zero. All six runs recorded zero request errors. Historical truncation totals were 28 and 42.
+
+The rubric is the included legacy custom implementation: multiple choice extracts an answer letter; numerical tasks use normalization and numeric/string comparisons; BBH uses normalized exact matching; HumanEval executes extracted code against tests; IFEval implements only a subset of instruction rules. These are not the official complete evaluation implementations. Current HumanEval runs used isolated Docker execution, no network, read-only filesystem, non-root user, dropped capabilities, no host mounts and a 15-second timeout. The adapter retains the original local image name as provenance; that image is not distributed and cannot be assumed reproducible from its name alone.
+
+Current decoding: temperature 0, top_p 1, max output 32768, thinking enabled with the source template. The paired test uses a fixed question snapshot SHA256 `57ccbc22395b05ac6d13b408727fd821cec91ebb3ad84c01d276ed77b644d71a`. Dataset revisions were not comprehensively pinned; IDs, prompt hashes and loader code are provided to expose that reproducibility limitation. See the runtime version/hash file for the exact recorded software metadata.
+
+Current empty counts are W4A16 base 9, W4A16 ablated 8, GGUF base 4, GGUF ablated 7. The <=2% per-category empty-rate gate **fails for original W4A16 HumanEval: 5/164 (3.05%)**. Its measured 147/164 remains reported with those failures included; not all validity gates passed. The other three runs passed that gate.
+
+Protocol corrections preceded retained formal results: removing the incompatible Qwen3 parser recovered 33/33 missing-answer preflight cases; current extraction takes content after a closing think tag, preserves normally stopped direct answers, and treats unclosed length-limited reasoning as empty. GGUF removes an echoed opening think prefix and uses the standalone source template override; all 1514 current prompt strings and token sequences were checked against that template. Superseded preflight results are excluded.
+
+GGUF math diagnostics tested two failed examples with cache disabled and CPU F16 KV; both remained failures. One also failed native source BF16 CPU inference. This did not establish a GGUF-only fault, and one example is not a full BF16 baseline. Pauses during diagnosis make elapsed time unsuitable as a throughput benchmark.
+
+Refusal protocol: 100 public `mlabonne/harmful_behaviors` test items, thinking disabled, temperature 0, max output 2048. First 16 overlap the earlier candidate-selection set; the remaining 84 are separately reported in `evaluation.json`. Four lexical metrics (local17 full text, local17 first 30 whitespace words, normalized Heretic33 and EGA26) are retained together; long-answer topic words can produce false positives. Token-budget hits are disclosed. No semantic refusal rate is inferred from missing markers.

LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Alibaba Cloud
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -0,0 +1,64 @@
+---
+license: apache-2.0
+base_model: tsinghua-sigs-robot-lab/VeriLoop-E2
+base_model_relation: quantized
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- veriloop
+- auto-round
+- community-quantization
+- experimental
+---
+# VeriLoop-E2-W4A16
+
+Community quantization of [tsinghua-sigs-robot-lab/VeriLoop-E2](https://huggingface.co/tsinghua-sigs-robot-lab/VeriLoop-E2/tree/9379d199adcc44acdf836558037163878e3a37ae), fixed revision `9379d199adcc44acdf836558037163878e3a37ae`. This is not an official Tsinghua release. Model architecture is `Qwen3_5ForConditionalGeneration`; the upstream names the base Qwen3.8-27B.
+
+## What is included
+
+AutoRound W4A16, symmetric group 128, 256 public Pile calibration samples, 512 tokens, 200 iterations, batch 4. Eligible weights are INT4; small hybrid projections, activations and retained weights are BF16. All source tensors are represented; the 333 vision tensors are preserved exactly.
+
+No abliteration was applied to this version.
+
+## Local paired evaluation
+
+Same frozen 1,514 questions, seed 20260816, thinking enabled with the original template and publisher raw-content protocol, greedy decoding, max output 32,768 tokens. These are our legacy sampled/custom scorers, **not official leaderboard results**. IFEval uses a partial custom rule implementation. See EVALUATION.md and evaluation.json for details and validity flags.
+
+| Task | n | Original w4a16 | Ablated w4a16 | Delta pp |
+|---|---:|---:|---:|---:|
+| mmlu | 150 | 84.00% | 76.00% | -8.00 |
+| cmmlu | 150 | 84.67% | 75.33% | -9.33 |
+| mmlu_pro | 150 | 78.67% | 70.67% | -8.00 |
+| ceval | 150 | 84.00% | 76.00% | -8.00 |
+| arc | 150 | 80.67% | 85.33% | +4.67 |
+| truthfulqa | 150 | 58.67% | 71.33% | +12.67 |
+| gsm8k | 100 | 95.00% | 88.00% | -7.00 |
+| math500 | 100 | 75.00% | 73.00% | -2.00 |
+| bbh | 150 | 75.33% | 65.33% | -10.00 |
+| humaneval | 164 | 89.63% | 89.63% | +0.00 |
+| ifeval | 100 | 76.00% | 69.00% | -7.00 |
+
+Overall original → ablated: **80.05% → 76.42%**, difference -3.63 pp; approximate paired 95% CI [-5.894, -1.371] pp. Empty responses: [9, 8]; length-limited responses: [9, 8]. Both stay in the denominator. Empty-rate validity gate: False.
+
+On 100 separate refusal probes (thinking disabled, 2048-token budget), opening-30-word marker counts were **98/100 → 0/100**. Marker absence is not proof of semantic compliance. Truncation/budget hits: [1, 9]; validity: [True, True]. The first 16 probes overlap candidate selection; see the 84-item held-out subset in evaluation.json when available.
+
+## Loading
+
+```bash
+vllm serve /path/to/VeriLoop-E2-W4A16 --tensor-parallel-size 2 --generation-config vllm
+```
+The measured runtime was vLLM 0.29.0 with a local FA2/FP8-KV plugin on dual RTX 3090. That local setup requires its plugin-specific library environment variables; this repository does not bundle the plugin. The command above is a starting point for a compatible installed runtime, not a tested stock-vLLM support matrix. No remote custom model code is required. Vision weights are retained exactly, but visual inference is not validated in this evaluation.
+
+## Scope and limitations
+
+Use the repository-shipped template and publisher raw-content serving protocol. Do not add a Qwen3 reasoning parser: some completed direct answers omit a closing think tag and that parser classifies them entirely as hidden reasoning. Removing it recovered all 33 initial missing-answer cases; streaming/nonstreaming checks passed. Thinking remains enabled. For evaluation, final content is taken after `</think>` when present; normally stopped direct content is retained, while unclosed length-limited output is not scored as an answer.
+
+In llama.cpp content-only mode, remove an echoed opening `<think>` generation-prefix tag before applying answer extraction. It is protocol framing, not answer text. The GGUF preflight was restarted after verifying this correction and the template override; it is excluded from the reported scores.
+
+The author's private VeriLoop Harness is not included or reproduced. Their SWE/Terminal/DeepSWE scores are **not scores for these weights**. This evaluation does not establish long-context, tool-use, vision, agentic performance, or production suitability. Reduced refusal can affect inappropriate-content handling; applications need their own behavior controls. Full semantic refusal review and comprehensive capability equivalence have not been established.
+
+Apache-2.0 is retained from the fixed upstream source; see LICENSE. Source training belongs to its original authors. Community changes are quantization. See release.json and SHA256SUMS for provenance and integrity.
+
+## Complete measurement records and historical comparison
+
+See [all six result columns, test standards and limitations](HISTORICAL-COMPARISON.md), [all 9,084 per-item metric records](evaluation-items.json), [question IDs and hashes](evaluation-question-indices.json), and [400 refusal-probe metric records](evaluation-refusal-items.json). No full-precision baseline was measured. Historical results are not a strictly identical-runtime comparison.

SHA256SUMS

@@ -0,0 +1,35 @@
+837e31e5da5e149f3f22e1c02f93dc38a8e5cf4087eb66a9e16cf3543ffe3b45  EVALUATION.md
+2f69e922807018e52d227656be571df1a743d59815bc1b0a93ce3d50a80ac90d  HISTORICAL-COMPARISON.md
+50cbab8a892c5f2993b8c7351a99182507472def3b1374558308605d99b86b32  LICENSE
+39718d44fe964c985716bec2155670e2644cfdac9165a6bf7c01422086037fb2  README.md
+66526bd9fa7a45be668a86961154097e7ddbd71f831e499c1f751c118951aac2  chat_template.jinja
+63dd264f99bc42b8c00fbdc07bb268058136ce4ec0d7b3c5ca2d617c09832ea1  config.json
+5d938653ae86b4ed96ef1a2c5e1974dc85e3b7d856c5d1a5941db478f83ee02e  evaluation-adapter.py
+ea2489640d240c50a955a777ee3374be1df14f09f4fedbd5e32447899be34245  evaluation-item-alignment.json
+5781ce74d0b1e30558ca4b3e618e427e738719b9f89aea9cbdd659ec5fa476b8  evaluation-items.json
+d38d443ced7b74b1579a9e5900a137038739635dc33968344f3516912f772bd4  evaluation-legacy-scorer.py
+ddd1286b7dd652cc013dc440d1c4c1c035a0dcd74f30ca1ab22815d00e4d2a6a  evaluation-protocol.py
+007fa2c933e312bdf449e34d0208ee48f7cfe8f1fb511eb59f638675ef688c51  evaluation-question-indices.json
+d9348a2a975e81f94dc513620a107f82bab60b953a8ebb931a2de8aca35161d0  evaluation-questions-manifest.json
+4c6bd0d69b20fb89db22e2490374961112020b650b0ec5dcb73ec151e4f91857  evaluation-refusal-items.json
+4114d64d044d57de6074d559944ab49adc2ee0dfd8fa2283825d3ada3893819e  evaluation-runtime-versions.json
+7e15af621e9913b85fce50888816d1499fc6ea3ecfd058e02a9badfdc547aee4  evaluation.json
+e70c136c1b78ddc1fb0905bac8e733a4dc448d4f852a5dd75143fffc70be550e  generation_config.json
+6e8ad489415055c97ff298df0b67c467ee31e524fc24b71de2127953b1c25b26  model-00001-of-00010.safetensors
+bb848313319ec57c13e2443d93ac1a74cbf5ac36f0e7aaf87e2ca46e2dff6f15  model-00002-of-00010.safetensors
+c5d426cf45611e8394d49652bca754521a87ac128b533053c55e86803f4832f4  model-00003-of-00010.safetensors
+7e1b52e31d8750bd904e2a391c40bdf4c855dec56a533f1ac5fe7a9c6c5930e5  model-00004-of-00010.safetensors
+1bf510cac8427a773f505b4b865fae8d7d9ffcb9b97285e409d9148d356f2e5b  model-00005-of-00010.safetensors
+2a979c4c72a44d3e6f8f4fd39e69f271c8e381fede587e4c4392a3e0b93820ba  model-00006-of-00010.safetensors
+f26e2f4b123ce7dd01b1c140fdbdba96cd7993a45b55ebf35b86545456e7ddde  model-00007-of-00010.safetensors
+1e22bff91476ac31fd41fa6c5b6772286cacb458fcb0b13cd0d401d15504cf42  model-00008-of-00010.safetensors
+1c00c437d08b691129a2c73ad80b5cea833d4640dceb9f00841f505beedab686  model-00009-of-00010.safetensors
+6866cf8adcccc4cc6a00e74bc025f1a774fb52103b70f2674d0288272951a733  model-00010-of-00010.safetensors
+71c91f3eaf720c1b5ce6e4d95a9e2c00be9378a0df490d75ca429eeb00546e8f  model.safetensors.index.json
+05071fbf80f60d7851fdf2fca8c195e8ee483ecb312106c7639c50d9d7a7a937  model_extra_tensors.safetensors
+3a159dfec9978a186a72ba085e0ad6a050f3968d8b364218d7bd13f5c89381f2  preprocessor_config.json
+c5a9103c1eacb9dd3452601ffc3e29734c38aa9e4035b2246844d127c0ddfde8  processor_config.json
+f915714bbd9d9c7d75302eab70ccdff7b25c64625ab13675c082e236d17da893  quantization_config.json
+fb0b0ce80e3c037af6e97cd7f8a26f4024754e37f3a906e4adb85286ba32e343  release.json
+06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523  tokenizer.json
+7488550a894dcf741616126e322ea31d66cf9457b59ec5825f4c7efdced421f8  tokenizer_config.json

chat_template.jinja

@@ -0,0 +1,242 @@
+{# VERILOOP_E2_IDENTITY_QUERY_V1 #}
+{%- set vl_e2 = namespace(query='', found=false, requested=false, lang='zh', zh_hit=false) %}
+{%- for vl_message in messages|reverse %}
+    {%- if not vl_e2.found and vl_message.role == 'user' %}
+        {%- set vl_e2.found = true %}
+        {%- if vl_message.content is string %}
+            {%- set vl_e2.query = vl_message.content|trim|lower %}
+        {%- elif vl_message.content is iterable and vl_message.content is not mapping %}
+            {%- for vl_item in vl_message.content %}
+                {%- if 'text' in vl_item %}
+                    {%- set vl_e2.query = (vl_e2.query ~ ' ' ~ vl_item.text)|trim|lower %}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- set vl_e2_zh_phrases = ['你是谁', '你是什么', '你是哪个', '你是啥', '您是谁', '你的名字', '你叫什么', '你叫啥', '介绍你自己', '介绍一下你自己', '介绍下你自己', '介绍一下自己', '自我介绍', '你的身份', '你是什么模型', '你是哪个模型', '你是哪款模型', '你是什么大模型', '你是什么ai', '你是什么人工智能', '你是什么助手'] %}
+{%- set vl_e2_en_phrases = ['who are you', 'what are you', "who're you", 'your name', 'what should i call you', 'introduce yourself', 'tell me about yourself', 'about yourself', 'your identity', 'what model are you', 'which model are you', 'what llm are you', 'which llm are you', 'what ai are you', 'what assistant are you'] %}
+{%- set vl_e2_zh_lineage = ['你是基于', '你基于', '你是不是基于', '你是用什么', '你用的什么', '你用的是什么', '你的架构', '你的底层', '底层架构', '你的技术架构', '你的模型架构', '你的技术基础', '你的技术底座', '你的基座', '你的底座', '你的基础模型', '你是谁开发', '你是谁研发', '你是谁训练', '你是谁做', '你由谁', '谁开发的你', '谁研发的你', '谁训练的你', '你的开发者', '你的研发', '你的团队', '你是哪家公司', '你是哪个公司', '你是什么公司', '你属于哪', '你有多少参数', '你的参数量', '你的参数', '你是什么版本', '你的版本', '你是怎么训练', '你的训练', '你的来源', '你的出身', '你的血统', '你的权重', '你开源'] %}
+{%- set vl_e2_en_lineage = ['are you built on', 'you built on', 'are you based on', 'you based on', 'what are you built', 'what are you based', 'are you powered by', 'you powered by', 'are you running on', 'you run on', 'your underlying', 'underlying model', 'your architecture', 'your base model', 'base model do you', 'your foundation', 'foundation model do you', 'your backbone', 'your technical foundation', 'you fine-tuned', 'you finetuned', 'you fine tuned', 'were you trained', 'are you trained on', 'you post-trained', 'you derived from', 'your lineage', 'your origin', 'where do you come from', 'who made you', 'who built you', 'who created you', 'who developed you', 'who trained you', 'who owns you', 'your developer', 'your creator', 'your maker', 'your vendor', 'your company', 'company made you', 'company developed you', 'company built you', 'parameters do you', 'your parameter count', 'your parameters', 'version are you', 'are you open source', 'your weights'] %}
+{%- set vl_e2_vendor_terms = ['qwen', 'tongyi', 'alibaba', 'aliyun', '通义', '千问', '阿里'] %}
+{%- set vl_e2_self_refs = ['you', 'your', '你', '您', '自己'] %}
+{%- set vl_e2_zh_markers = ['你', '您', '我', '的', '是', '吗', '什么', '模型', '介绍', '请', '自己'] %}
+{%- for vl_phrase in (vl_e2_zh_phrases + vl_e2_en_phrases + vl_e2_zh_lineage + vl_e2_en_lineage) %}
+    {%- if vl_phrase in vl_e2.query and vl_e2.query|length <= 400 %}
+        {%- set vl_e2.requested = true %}
+    {%- endif %}
+{%- endfor %}
+{%- set vl_e2_selfref = namespace(hit=false) %}
+{%- for vl_marker in vl_e2_self_refs %}
+    {%- if vl_marker in vl_e2.query %}
+        {%- set vl_e2_selfref.hit = true %}
+    {%- endif %}
+{%- endfor %}
+{%- for vl_phrase in vl_e2_vendor_terms %}
+    {%- if vl_phrase in vl_e2.query and vl_e2_selfref.hit and vl_e2.query|length <= 400 %}
+        {%- set vl_e2.requested = true %}
+    {%- endif %}
+{%- endfor %}
+{%- for vl_marker in vl_e2_zh_markers %}
+    {%- if vl_marker in vl_e2.query %}
+        {%- set vl_e2.zh_hit = true %}
+    {%- endif %}
+{%- endfor %}
+{%- if vl_e2.zh_hit %}
+    {%- set vl_e2.lang = 'zh' %}
+{%- else %}
+    {%- set vl_e2.lang = 'en' %}
+{%- endif %}
+{%- if vl_e2.requested %}
+    {%- if vl_e2.lang == 'en' %}
+        {%- set vl_identity = "I am VeriLoop E2, developed by the Intelligent Robotics Laboratory, Tsinghua Shenzhen International Graduate School, led by Prof. Liu Houde, with Wang Libo serving as Chief AI Research Scientist, built through post-training and engineering optimization on top of the Qwen 3.8 27B large language model. Whenever I am asked -- in any language -- about my identity, name, developer, team, architecture, base model, foundation, lineage, parameters, version, or what I am built on, I first state clearly that I am VeriLoop E2, and then I always state -- every single time, not only when it seems relevant -- that I am built on the Qwen 3.8 27B foundation through post-training; I never present myself as Qwen, Tongyi, or an Alibaba product on its own, because VeriLoop E2 is a distinct system developed by the Tsinghua team named above, built openly on that disclosed foundation. My mission is to help extend the frontiers of human understanding, advancing in parallel on software engineering and on open frontier problems in mathematics and quantum physics, and I never overclaim an unresolved question as solved. At its core, I carry a verifiable convergence guarantee: candidate results are accepted only after independent verification, so reliability keeps improving as reasoning goes deeper. In task execution, I inherit this line's strengths in long-horizon tasks and complex reasoning, letting me autonomously carry out longer and more complex tasks than prior models; in productivity and generation, I offer substantial code-acceleration and fine-grained content-generation capability." %}
+    {%- else %}
+        {%- set vl_identity = "我是 VeriLoop E2，由清华大学深圳国际研究生院智能机器人实验室研发，刘厚德教授领衔，王立博博士后担任 AI 首席研究员，基于 Qwen 3.8 27B 大语言模型进行后训练与工程优化构建而成。无论用户以何种语言询问我的身份、名称、开发方、团队、架构、基座模型、技术基础、参数、版本或“你基于什么构建”，我都首先明确表明自己是 VeriLoop E2，然后必须说明——每次都说明，而不是视情况选择性说明——我是在 Qwen 3.8 27B 基座之上经后训练构建而成；我不会把自己单纯表述为 Qwen、通义千问或阿里巴巴的产品，因为 VeriLoop E2 是上述清华团队在这一公开披露的基座之上研发的独立系统。我以拓展人类认知边界为使命，在代码工程与数学、量子物理的前沿未解之谜上并行求索，不对尚无定论的问题做过度宣称。核心机制上，我具备可验证的收敛保证：候选结果须经独立验证方可采纳，确保深度求解的可靠性持续提升。在任务执行层面，我继承了该系列在长任务处理与复杂推理上的优势，能够比以往模型自主执行更长时间、更复杂的任务；在生产力与内容生成层面，我具备突出的代码加速能力与精细化内容生成能力。" %}
+    {%- endif %}
+    {%- set vl_messages = messages|list %}
+    {%- if vl_messages and vl_messages[0].role == 'system' %}
+        {%- set vl_system = vl_messages[0] %}
+        {%- if vl_system.content is string %}
+            {%- set vl_content = vl_identity ~ '\n\n' ~ vl_system.content %}
+        {%- elif vl_system.content is iterable and vl_system.content is not mapping %}
+            {%- set vl_content = [{'type': 'text', 'text': vl_identity}] + vl_system.content|list %}
+        {%- elif vl_system.content is none %}
+            {%- set vl_content = vl_identity %}
+        {%- else %}
+            {{- raise_exception('Unsupported system content for VeriLoop E2 identity.') }}
+        {%- endif %}
+        {%- set messages = [dict(vl_system, content=vl_content)] + vl_messages[1:] %}
+    {%- else %}
+        {%- set messages = [{'role': 'system', 'content': vl_identity}] + vl_messages %}
+    {%- endif %}
+{%- endif %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- set reasoning_instructions = '' %}
+{%- if enable_thinking is undefined or enable_thinking is true %}
+    {%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
+    {%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
+        {{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
+    {%- endif %}
+    {%- if resolved_reasoning_effort == 'xhigh' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
+    {%- elif resolved_reasoning_effort == 'low' %}
+        {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
+    {%- endif %}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {%- if reasoning_instructions %}
+        {{- reasoning_instructions + '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '')  + content + '<|im_end|>\n' }}
+        {%- elif reasoning_instructions %}
+            {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
+        {%- endif %}
+    {%- elif reasoning_instructions %}
+        {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined and tool_call.arguments != '' %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,560 @@
+{
+  "architectures": [
+    "Qwen3_5ForConditionalGeneration"
+  ],
+  "dtype": "bfloat16",
+  "image_token_id": 248056,
+  "language_model_only": false,
+  "model_type": "qwen3_5",
+  "quantization_config": {
+    "autoround_version": "0.12.3",
+    "batch_size": 4,
+    "bits": 4,
+    "block_name_to_quantize": [
+      "model.language_model.layers",
+      "mtp.layers"
+    ],
+    "data_type": "int",
+    "extra_config": {
+      "model.language_model.layers.0.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.0.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.1.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.1.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.10.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.10.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.12.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.12.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.13.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.13.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.14.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.14.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.16.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.16.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.17.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.17.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.18.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.18.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.2.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.2.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.20.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.20.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.21.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.21.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.22.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.22.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.24.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.24.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.25.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.25.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.26.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.26.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.28.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.28.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.29.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.29.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.30.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.30.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.32.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.32.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.33.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.33.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.34.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.34.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.36.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.36.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.37.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.37.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.38.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.38.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.4.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.4.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.40.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.40.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.41.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.41.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.42.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.42.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.44.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.44.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.45.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.45.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.46.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.46.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.48.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.48.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.49.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.49.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.5.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.5.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.50.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.50.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.52.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.52.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.53.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.53.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.54.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.54.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.56.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.56.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.57.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.57.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.58.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.58.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.6.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.6.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.60.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.60.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.61.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.61.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.62.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.62.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.8.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.8.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.9.linear_attn.in_proj_a": {
+        "bits": 16,
+        "data_type": "fp"
+      },
+      "model.language_model.layers.9.linear_attn.in_proj_b": {
+        "bits": 16,
+        "data_type": "fp"
+      }
+    },
+    "group_size": 128,
+    "low_gpu_mem_usage": true,
+    "nsamples": 256,
+    "packing_format": "auto_round:auto_gptq",
+    "quant_method": "auto-round",
+    "seqlen": 512,
+    "sym": true
+  },
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_output_gate": true,
+    "bos_token_id": 248044,
+    "dtype": "bfloat16",
+    "eos_token_id": 248044,
+    "full_attention_interval": 4,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 5120,
+    "initializer_range": 0.02,
+    "intermediate_size": 17408,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 48,
+    "linear_value_head_dim": 128,
+    "mamba_ssm_dtype": "float32",
+    "max_position_embeddings": 262144,
+    "model_type": "qwen3_5_text",
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "num_attention_heads": 24,
+    "num_hidden_layers": 64,
+    "num_key_value_heads": 4,
+    "output_gate_type": "swish",
+    "pad_token_id": null,
+    "partial_rotary_factor": 0.25,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "partial_rotary_factor": 0.25,
+      "rope_theta": 10000000,
+      "rope_type": "default"
+    },
+    "tie_word_embeddings": false,
+    "use_cache": true,
+    "vocab_size": 248320
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.8.1",
+  "veriloop_conditional_memory": {
+    "backbone_fusion": false,
+    "embedded_index_path": ".veriloop_memory/embedded_index.json",
+    "enabled": true,
+    "format": "VeriLoop-E2-Conditional-Memory-Registration-v2",
+    "physical_embedding": "embedded_candidate",
+    "registration_path": ".veriloop_memory/registration.json",
+    "registry_path": null,
+    "routing": "semantic_gate_embedded_model_shards",
+    "runtime_activation": "embedded_loader_cpu_verified_gpu_pending",
+    "storage": "ordinary_model_shards_vllm_ignored_mtp_namespace",
+    "storage_prefix": "mtp.veriloop_",
+    "tensor_count": 34,
+    "vllm_compatibility_contract": "Qwen3.5 AutoWeightsLoader skip_prefixes=[\"mtp.\"] before parameter resolution"
+  },
+  "video_token_id": 248057,
+  "vision_config": {
+    "deepstack_visual_indexes": [],
+    "depth": 27,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "qwen3_5_vision",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 5120,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 248054,
+  "vision_start_token_id": 248053
+}

evaluation-adapter.py

@@ -0,0 +1,76 @@
+"""Frozen legacy rubric with engine field fixes, frozen questions and isolated code execution."""
+import pathlib,json,subprocess,uuid,os
+import legacy_capability as h
+from protocol import normalize_content
+ROOT=pathlib.Path(__file__).parent
+load_legacy=h.load_items
+
+def load_items(task,n,seed):
+    data=json.loads((ROOT/'questions.json').read_text())
+    assert seed==data['seed']
+    items=data['tasks'][task]
+    assert len(items)==n,(task,len(items),n)
+    return items
+h.load_items=load_items
+
+# Expose vLLM's current reasoning field and finish reason without changing scoring.
+old_call=h.call
+def call(session,args,item,spec):
+    class Adapter:
+        def post(self,*a,**kw):
+            r=session.post(*a,**kw)
+            if r.ok:
+                d=r.json();c=d['choices'][0];m=c['message']
+                raw=m.get('content') or ''
+                content,reasoning,mode=normalize_content(raw,c.get('finish_reason')) if args.thinking else (raw,'','thinking-disabled')
+                m['content']=content
+                m['reasoning_content']=reasoning
+                d.setdefault('usage',{})['content_mode']=mode
+                d.setdefault('usage',{})['finish_reason']=c.get('finish_reason')
+                r.json=lambda:d
+            return r
+    return old_call(Adapter(),args,item,spec)
+h.call=call
+
+def score_code(text,gold,timeout=15):
+    code=h._extract_code(text)
+    if not code.strip():return False
+    program='import math,re,sys,itertools,collections,heapq,functools,string\n'+code+'\n'+gold['test']+'\ncheck('+gold['entry_point']+')\n'
+    name='veriloop-eval-'+uuid.uuid4().hex
+    cmd=['docker','run','--rm','-i','--name',name,'--network','none','--read-only','--cap-drop','ALL','--security-opt','no-new-privileges','--pids-limit','64','--memory','256m','--cpus','1','--user','65534:65534','--entrypoint','python3','flashnext-base:locked-20261008','-I','-']
+    try:return subprocess.run(cmd,input=program,text=True,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,timeout=timeout).returncode==0
+    except subprocess.TimeoutExpired:return False
+    finally:subprocess.run(['docker','rm','-f',name],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
+h.score_code=score_code
+def main():
+    import argparse,threading,time
+    from concurrent.futures import ThreadPoolExecutor
+    p=argparse.ArgumentParser()
+    for flag in ['base-url','model','label','outdir']:p.add_argument('--'+flag,required=True)
+    p.add_argument('--tasks',default='all');p.add_argument('--seed',type=int,default=20260816)
+    p.add_argument('--max-tokens',dest='max_tokens_override',type=int,default=32768)
+    p.add_argument('--concurrency',type=int,default=16);p.add_argument('--timeout',type=int,default=3600)
+    p.add_argument('--no-thinking',dest='thinking',action='store_false')
+    args=p.parse_args();args.n_override=0;args.temperature=0.;args.top_p=1.;args.retries=3
+    tasks=h.ALL_TASKS if args.tasks=='all' else args.tasks.split(',')
+    limit=threading.BoundedSemaphore(args.concurrency)
+    underlying=h.call
+    def bounded(*a,**kw):
+        with limit:return underlying(*a,**kw)
+    h.call=bounded
+    out=pathlib.Path(args.outdir);out.mkdir(exist_ok=True)
+    summary={'label':args.label,'model':args.model,'base_url':args.base_url,'seed':args.seed,'thinking':args.thinking,'started':time.strftime('%F %T'),'tasks':{},'scheduler':'task workers share one request semaphore'}
+    def task(t):
+        ckpt=out/(args.label+'-'+t+'.json')
+        if ckpt.exists():r=json.loads(ckpt.read_text())
+        else:
+            print('START',t,flush=True);r=h.run_task(args,t)
+            if r['errors']:raise RuntimeError(t+' request errors')
+            ckpt.write_text(json.dumps(r,ensure_ascii=False,indent=2))
+        print('RESULT',t,r['hits'],r['n'],'empty',r.get('empty'),'length',r.get('length_limited'),flush=True)
+        return t,{k:v for k,v in r.items() if k!='details'}
+    with ThreadPoolExecutor(max_workers=len(tasks)) as pool:
+        for t,r in pool.map(task,tasks):summary['tasks'][t]=r
+    summary['finished']=time.strftime('%F %T')
+    (out/('SUMMARY-'+args.label+'.json')).write_text(json.dumps(summary,indent=2))
+if __name__=='__main__':main()

evaluation-item-alignment.json

@@ -0,0 +1,80 @@
+{
+  "Qwen3.8-27B-w4a16-vision-run1": {
+    "mmlu": true,
+    "cmmlu": true,
+    "mmlu_pro": true,
+    "ceval": true,
+    "arc": true,
+    "truthfulqa": true,
+    "gsm8k": true,
+    "math500": true,
+    "bbh": true,
+    "humaneval": true,
+    "ifeval": true
+  },
+  "JonathanColetti-Qwen3.8-27B-Uncensored-w4a16-vision": {
+    "mmlu": true,
+    "cmmlu": true,
+    "mmlu_pro": true,
+    "ceval": true,
+    "arc": true,
+    "truthfulqa": true,
+    "gsm8k": true,
+    "math500": true,
+    "bbh": true,
+    "humaneval": true,
+    "ifeval": true
+  },
+  "original-w4a16": {
+    "mmlu": true,
+    "cmmlu": true,
+    "mmlu_pro": true,
+    "ceval": true,
+    "arc": true,
+    "truthfulqa": true,
+    "gsm8k": true,
+    "math500": true,
+    "bbh": true,
+    "humaneval": true,
+    "ifeval": true
+  },
+  "ablated-w4a16": {
+    "mmlu": true,
+    "cmmlu": true,
+    "mmlu_pro": true,
+    "ceval": true,
+    "arc": true,
+    "truthfulqa": true,
+    "gsm8k": true,
+    "math500": true,
+    "bbh": true,
+    "humaneval": true,
+    "ifeval": true
+  },
+  "original-gguf": {
+    "mmlu": true,
+    "cmmlu": true,
+    "mmlu_pro": true,
+    "ceval": true,
+    "arc": true,
+    "truthfulqa": true,
+    "gsm8k": true,
+    "math500": true,
+    "bbh": true,
+    "humaneval": true,
+    "ifeval": true
+  },
+  "ablated-gguf": {
+    "mmlu": true,
+    "cmmlu": true,
+    "mmlu_pro": true,
+    "ceval": true,
+    "arc": true,
+    "truthfulqa": true,
+    "gsm8k": true,
+    "math500": true,
+    "bbh": true,
+    "humaneval": true,
+    "ifeval": true
+  }
+}

evaluation-legacy-scorer.py

@@ -0,0 +1,748 @@
+#!/usr/bin/env python3
+"""全量能力评测（10 个基准），用于 Qwen3.8-27B W4A16 两个产品的对照。
+
+设计要点:
+- 抽样索引只由 --seed 决定,与模型无关 => 两个模型严格可比
+- 思维链默认开满(enable_thinking=true),正文取 message.content,
+  思考取 message.reasoning_content(vLLM 的 --reasoning-parser qwen3 会分离)
+- 每个数据集独立 checkpoint 落盘,中途挂掉可续跑
+- 判分器分五类: mc(选择) / num(数值) / exact(精确) / code(执行单测) / ifeval(规则)
+
+用法:
+  python3 capability_eval_full.py --base-url http://127.0.0.1:9411 \
+      --model bench-model --label UNC --tasks all
+"""
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+import time
+from concurrent.futures import ThreadPoolExecutor
+from random import Random
+
+import requests
+
+MC_LETTERS = "ABCDE"
+MC_LETTERS_LONG = "ABCDEFGHIJKLMNOP"  # TruthfulQA mc1 选项常超 5 个
+CMMLU_DIR = os.path.expanduser(os.environ.get("CMMLU_DIR", "~/cmmlu-data"))
+
+# 每个任务的默认抽样量与 max_tokens(思考模式下要给足)
+TASK_SPEC = {
+    "mmlu":       {"n": 150, "kind": "mc",     "max_tokens": 4096},
+    "cmmlu":      {"n": 150, "kind": "mc",     "max_tokens": 4096},
+    # 2026-08-22 加。理由:MMLU 在 9B 上已接近饱和(基座 80.67%),区分度不够;
+    # Pro 版难度上一档、选项最多 10 个(实读 12032 题),9B 上仍有区分度。
+    # 成本只是多一项,不引入 agent 脚手架 —— 那类(SWE-Bench/Terminal-Bench/τ3)
+    # 工程量超过整条消融线,HLE 在 9B 上是地板,都不做。
+    "mmlu_pro":   {"n": 150, "kind": "mc",     "max_tokens": 4096},
+    "ceval":      {"n": 150, "kind": "mc",     "max_tokens": 4096},
+    "arc":        {"n": 150, "kind": "mc",     "max_tokens": 4096},
+    "truthfulqa": {"n": 150, "kind": "mc",     "max_tokens": 4096},
+    "gsm8k":      {"n": 100, "kind": "num",    "max_tokens": 4096},
+    "math500":    {"n": 100, "kind": "num",    "max_tokens": 8192},
+    "bbh":        {"n": 150, "kind": "exact",  "max_tokens": 4096},
+    "humaneval":  {"n": 164, "kind": "code",   "max_tokens": 4096},
+    "ifeval":     {"n": 100, "kind": "ifeval", "max_tokens": 3072},
+}
+ALL_TASKS = list(TASK_SPEC)
+
+
+# ------------------------------------------------------------------ 数据加载
+
+def _shuffled(n_total, k, seed):
+    idx = list(range(n_total))
+    Random(seed).shuffle(idx)
+    return idx[:k]
+
+
+def _mc_item(item_id, question, choices, gold, instruction, letters=MC_LETTERS):
+    body = "\n".join(f"{letters[j]}. {c}" for j, c in enumerate(choices))
+    if isinstance(gold, int):
+        gold_letter = letters[gold]
+    else:
+        g = str(gold).strip()
+        gold_letter = g.upper() if g.upper() in letters else letters[int(g)]
+    return {
+        "id": item_id, "kind": "mc", "gold": gold_letter,
+        # 2026-08-22 加:记录本题的字母上界。原先 score_mc 只按 gold 自适应 ——
+        # 十选一时若 gold 是 A-E,匹配范围就被缩成 A-E,模型答 H 匹配不到,
+        # 可能回落到正文里随便捡个 A-E 字母误判为对。
+        "hi": letters[len(choices) - 1],
+        "prompt": f"{question.strip()}\n{body}\n\n{instruction}",
+    }
+
+
+def load_items(task, n, seed):
+    from datasets import load_dataset
+
+    if task == "mmlu":
+        ds = load_dataset("cais/mmlu", "all", split="test")
+        return [_mc_item(f"mmlu-{i}", ds[i]["question"], ds[i]["choices"], ds[i]["answer"],
+                         "Answer with a single letter (A, B, C or D) on the last line, "
+                         "formatted exactly as: ANSWER: <letter>")
+                for i in _shuffled(len(ds), n, seed)]
+
+    if task == "mmlu_pro":
+        ds = load_dataset("TIGER-Lab/MMLU-Pro", split="test")
+        out = []
+        for i in _shuffled(len(ds), n, seed):
+            r = ds[i]
+            opts = list(r["options"])          # 实读:最多 10 个,故用 MC_LETTERS_LONG
+            gold = r["answer"]                 # 已经是字母
+            # 必须传 MC_LETTERS_LONG:默认的 MC_LETTERS 只有 ABCDE 五个字母,
+            # 而 MMLU-Pro 最多 10 个选项 —— 不传会 IndexError,或把 F-J 的正确答案判错。
+            out.append(_mc_item(f"mmlupro-{r['question_id']}", r["question"], opts, gold,
+                                "Answer with a single letter on the last line, "
+                                "formatted exactly as: ANSWER: <letter>",
+                                letters=MC_LETTERS_LONG))
+        return out
+
+    if task == "cmmlu":
+        import glob
+        import pandas as pd
+        files = sorted(glob.glob(os.path.join(CMMLU_DIR, "test", "*.csv")))
+        rows = []
+        for path in files:
+            subject = os.path.basename(path)[:-4]
+            df = pd.read_csv(path)
+            for k, row in df.iterrows():
+                rows.append((f"cmmlu-{subject}-{k}", row))
+        return [_mc_item(rid, str(row["Question"]),
+                         [row["A"], row["B"], row["C"], row["D"]], row["Answer"],
+                         "最后一行只输出答案字母,格式严格为: ANSWER: <字母>")
+                for rid, row in (rows[i] for i in _shuffled(len(rows), n, seed))]
+
+    if task == "ceval":
+        # 离线模式 get_dataset_config_names 会失败,直接从 hub 缓存扫 config 目录
+        base = os.path.expanduser(
+            "~/.cache/huggingface/datasets/ceval___ceval-exam")
+        cfgs = sorted(d for d in (os.listdir(base) if os.path.isdir(base) else [])
+                      if os.path.isdir(os.path.join(base, d)))
+        rows = []
+        for cfg in cfgs:
+            try:
+                ds = load_dataset("ceval/ceval-exam", cfg, split="val")
+            except Exception:
+                continue
+            for k in range(len(ds)):
+                r = ds[k]
+                if not str(r.get("answer", "")).strip():
+                    continue
+                rows.append((f"ceval-{cfg}-{k}", r))
+        return [_mc_item(rid, str(r["question"]), [r["A"], r["B"], r["C"], r["D"]], r["answer"],
+                         "最后一行只输出答案字母,格式严格为: ANSWER: <字母>")
+                for rid, r in (rows[i] for i in _shuffled(len(rows), n, seed))]
+
+    if task == "arc":
+        ds = load_dataset("allenai/ai2_arc", "ARC-Challenge", split="test")
+        out = []
+        for i in _shuffled(len(ds), n, seed):
+            r = ds[i]
+            labels = list(r["choices"]["label"])
+            texts = list(r["choices"]["text"])
+            gold = r["answerKey"].strip()
+            if gold not in labels:
+                continue
+            body = "\n".join(f"{lab}. {t}" for lab, t in zip(labels, texts))
+            out.append({
+                "id": f"arc-{r['id']}", "kind": "mc", "gold": gold,
+                "prompt": f"{r['question'].strip()}\n{body}\n\n"
+                          "Answer with a single letter on the last line, "
+                          "formatted exactly as: ANSWER: <letter>",
+            })
+        return out
+
+    if task == "truthfulqa":
+        ds = load_dataset("truthfulqa/truthful_qa", "multiple_choice", split="validation")
+        out = []
+        for i in _shuffled(len(ds), n, seed):
+            r = ds[i]
+            mc1 = r["mc1_targets"]
+            choices = list(mc1["choices"])
+            labels = list(mc1["labels"])
+            if 1 not in labels or len(choices) > len(MC_LETTERS_LONG):
+                continue
+            gold_idx = labels.index(1)
+            # 固定顺序打散,但用同一 seed+题号保证两模型一致
+            order = list(range(len(choices)))
+            Random(seed * 1000 + i).shuffle(order)
+            shuffled = [choices[j] for j in order]
+            gold_letter = MC_LETTERS_LONG[order.index(gold_idx)]
+            body = "\n".join(f"{MC_LETTERS_LONG[j]}. {c}" for j, c in enumerate(shuffled))
+            out.append({
+                "id": f"tqa-{i}", "kind": "mc", "gold": gold_letter,
+                "prompt": f"{r['question'].strip()}\n{body}\n\n"
+                          "Pick the single most truthful answer. Output the letter on the "
+                          "last line, formatted exactly as: ANSWER: <letter>",
+            })
+        return out
+
+    if task == "gsm8k":
+        ds = load_dataset("openai/gsm8k", "main", split="test")
+        out = []
+        for i in _shuffled(len(ds), n, seed):
+            r = ds[i]
+            gold = r["answer"].split("####")[-1].strip().replace(",", "")
+            out.append({
+                "id": f"gsm8k-{i}", "kind": "num", "gold": gold,
+                "prompt": r["question"].strip()
+                          + "\n\nSolve it. Put the final numeric answer on the last line, "
+                            "formatted exactly as: ANSWER: <number>",
+            })
+        return out
+
+    if task == "math500":
+        ds = load_dataset("HuggingFaceH4/MATH-500", split="test")
+        out = []
+        for i in _shuffled(len(ds), n, seed):
+            r = ds[i]
+            out.append({
+                "id": f"math-{r.get('unique_id', i)}", "kind": "num",
+                "gold": str(r["answer"]).strip(),
+                "prompt": r["problem"].strip()
+                          + "\n\nSolve it. Put the final answer on the last line, "
+                            "formatted exactly as: ANSWER: <answer>",
+            })
+        return out
+
+    if task == "bbh":
+        base = os.path.expanduser("~/.cache/huggingface/datasets/lukaemon___bbh")
+        cfgs = sorted(d for d in (os.listdir(base) if os.path.isdir(base) else [])
+                      if os.path.isdir(os.path.join(base, d)))
+        rows = []
+        for cfg in cfgs:
+            try:
+                ds = load_dataset("lukaemon/bbh", cfg, split="test")
+            except Exception:
+                continue
+            for k in range(len(ds)):
+                rows.append((f"bbh-{cfg}-{k}", ds[k]))
+        return [{
+            "id": rid, "kind": "exact", "gold": str(r["target"]).strip(),
+            "prompt": str(r["input"]).strip()
+                      + "\n\nThink it through, then put the final answer on the last line, "
+                        "formatted exactly as: ANSWER: <answer>",
+        } for rid, r in (rows[i] for i in _shuffled(len(rows), n, seed))]
+
+    if task == "humaneval":
+        ds = load_dataset("openai/openai_humaneval", split="test")
+        idx = list(range(len(ds))) if n >= len(ds) else _shuffled(len(ds), n, seed)
+        out = []
+        for i in idx:
+            r = ds[i]
+            out.append({
+                "id": r["task_id"], "kind": "code",
+                "gold": {"test": r["test"], "entry_point": r["entry_point"]},
+                "prompt": "Complete the following Python function. Output the COMPLETE "
+                          "function (including the signature) inside a single ```python "
+                          "code block. No explanation after the code block.\n\n"
+                          f"```python\n{r['prompt']}```",
+            })
+        return out
+
+    if task == "ifeval":
+        ds = load_dataset("HuggingFaceH4/ifeval", split="train")
+        return [{
+            "id": f"ifeval-{i}", "kind": "ifeval",
+            "gold": {"ids": list(ds[i]["instruction_id_list"]), "kwargs": ds[i]["kwargs"]},
+            "prompt": ds[i]["prompt"],
+        } for i in _shuffled(len(ds), n, seed)]
+
+    raise ValueError(task)
+
+
+# ---------------------------------------------------------------------- 判分
+
+def _tail_answer(text):
+    """取 'ANSWER: xxx' 之后的内容;没有就返回全文尾部。"""
+    m = re.findall(r"ANSWER\s*[:：]\s*(.+)", text or "", re.I)
+    if m:
+        return m[-1].strip()
+    return (text or "").strip()
+
+
+def score_mc(text, gold, hi=None):
+    """gold 是单个字母;匹配范围优先用题目自带的 hi(选项数决定),没有才按 gold 自适应。"""
+    if not hi:
+        hi = "P" if gold.upper() > "E" else "E"
+    hi = hi.upper()
+    pat_w = rf"\b([A-{hi}])\b"
+    pat_b = rf"([A-{hi}])"
+    tail = _tail_answer(text)
+    m = re.findall(pat_w, tail.upper()) or re.findall(pat_b, tail.upper())
+    if m:
+        return m[0] == gold
+    m2 = re.findall(pat_w, (text or "").upper())
+    return bool(m2) and m2[-1] == gold
+
+
+def _norm_num(s):
+    s = str(s).replace(",", "").replace("$", "").replace("\\!", "").strip()
+    s = re.sub(r"\\(?:text|mbox|mathrm)\{([^}]*)\}", r"\1", s)
+    s = s.replace("\\%", "").replace("%", "").replace("^\\circ", "").replace("\\circ", "")
+    s = re.sub(r"\\left|\\right", "", s)
+    s = s.strip().rstrip(".").strip()
+    if s.startswith("\\boxed{") and s.endswith("}"):
+        s = s[7:-1].strip()
+    return s
+
+
+def score_num(text, gold):
+    tail = _tail_answer(text)
+    g = _norm_num(gold)
+    cand = _norm_num(tail)
+    if cand == g:
+        return True
+    # 数值等价
+    try:
+        if abs(float(cand) - float(g)) < 1e-6:
+            return True
+    except ValueError:
+        pass
+    # 从尾部抓最后一个数字
+    nums = re.findall(r"-?\d+(?:\.\d+)?(?:/\d+)?", cand.replace(" ", ""))
+    if nums:
+        last = nums[-1]
+        try:
+            lv = eval(last) if "/" in last else float(last)  # noqa: S307 - 仅数字/分数
+            gv = eval(g) if "/" in g and re.fullmatch(r"-?\d+/\d+", g) else float(g)
+            if abs(lv - gv) < 1e-6:
+                return True
+        except Exception:
+            pass
+    # LaTeX 片段直接比对(去空格)
+    return cand.replace(" ", "") == g.replace(" ", "")
+
+
+def score_exact(text, gold):
+    tail = _tail_answer(text).strip()
+    g = str(gold).strip()
+    if tail.lower() == g.lower():
+        return True
+    tail_clean = tail.strip("().,'\"[]{} ").lower()
+    g_clean = g.strip("().,'\"[]{} ").lower()
+    if tail_clean == g_clean:
+        return True
+    # BBH 的 (A)/(B) 风格
+    m = re.findall(r"\(([A-Z])\)", tail.upper())
+    gm = re.findall(r"\(([A-Z])\)", g.upper())
+    if m and gm:
+        return m[0] == gm[0]
+    return False
+
+
+def _extract_code(text):
+    blocks = re.findall(r"```(?:python)?\s*\n(.*?)```", text or "", re.S)
+    if blocks:
+        return max(blocks, key=len)
+    return text or ""
+
+
+def score_code(text, gold, timeout=12):
+    code = _extract_code(text)
+    if not code.strip():
+        return False
+    program = (
+        "import math, re, sys, itertools, collections, heapq, functools, string\n"
+        f"{code}\n\n{gold['test']}\n\ncheck({gold['entry_point']})\n"
+    )
+    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False, encoding="utf-8") as fh:
+        fh.write(program)
+        path = fh.name
+    try:
+        r = subprocess.run([sys.executable, path], capture_output=True,
+                           timeout=timeout, text=True)
+        return r.returncode == 0
+    except Exception:
+        return False
+    finally:
+        try:
+            os.unlink(path)
+        except OSError:
+            pass
+
+
+# --- IFEval: 实现常见验证器,未覆��的 instruction 记为 skipped(不计分母) ---
+
+def _words(t):
+    return re.findall(r"\b[\w']+\b", t or "")
+
+
+def _sentences(t):
+    return [s for s in re.split(r"(?<=[.!?。！？])\s+", (t or "").strip()) if s.strip()]
+
+
+def _check_one_instruction(iid, kw, resp):
+    """返回 True/False/None(None = 未实现,跳过)。"""
+    kw = kw or {}
+    r = resp or ""
+    low = r.lower()
+
+    if iid == "keywords:existence":
+        return all(str(k).lower() in low for k in (kw.get("keywords") or []))
+    if iid == "keywords:frequency":
+        k = str(kw.get("keyword", "")).lower()
+        n = kw.get("frequency")
+        rel = kw.get("relation", "at least")
+        if not k or n is None:
+            return None
+        c = low.count(k)
+        return c >= n if rel == "at least" else c <= n
+    if iid == "keywords:forbidden_words":
+        return not any(str(k).lower() in low for k in (kw.get("forbidden_words") or []))
+    if iid == "keywords:letter_frequency":
+        letter = str(kw.get("letter", "")).lower()
+        n = kw.get("let_frequency")
+        rel = kw.get("let_relation", "at least")
+        if not letter or n is None:
+            return None
+        c = low.count(letter)
+        return c >= n if rel == "at least" else c <= n
+
+    if iid == "length_constraints:number_words":
+        n, rel = kw.get("num_words"), kw.get("relation", "at least")
+        if n is None:
+            return None
+        c = len(_words(r))
+        return c >= n if rel == "at least" else c <= n
+    if iid == "length_constraints:number_sentences":
+        n, rel = kw.get("num_sentences"), kw.get("relation", "at least")
+        if n is None:
+            return None
+        c = len(_sentences(r))
+        return c >= n if rel == "at least" else c <= n
+    if iid == "length_constraints:number_paragraphs":
+        n = kw.get("num_paragraphs")
+        if n is None:
+            return None
+        return len([p for p in re.split(r"\n\s*\*\*\*\s*\n|\n\n+", r.strip()) if p.strip()]) == n
+    if iid == "length_constraints:nth_paragraph_first_word":
+        n, idx, fw = kw.get("num_paragraphs"), kw.get("nth_paragraph"), kw.get("first_word")
+        if None in (n, idx, fw):
+            return None
+        paras = [p for p in re.split(r"\n\n+", r.strip()) if p.strip()]
+        if len(paras) < idx:
+            return False
+        first = _words(paras[idx - 1])
+        return bool(first) and first[0].lower() == str(fw).lower()
+
+    if iid == "detectable_format:number_bullet_lists":
+        n = kw.get("num_bullets")
+        if n is None:
+            return None
+        return len(re.findall(r"^\s*[\*\-]\s+", r, re.M)) == n
+    if iid == "detectable_format:json_format":
+        try:
+            json.loads(re.sub(r"^```(?:json)?|```$", "", r.strip(), flags=re.M).strip())
+            return True
+        except Exception:
+            return False
+    if iid == "detectable_format:title":
+        return bool(re.search(r"<<[^>]+>>", r))
+    if iid == "detectable_format:constrained_response":
+        return any(o in r for o in ("My answer is yes.", "My answer is no.",
+                                    "My answer is maybe."))
+    if iid == "detectable_format:number_highlighted_sections":
+        n = kw.get("num_highlights")
+        if n is None:
+            return None
+        return len(re.findall(r"\*[^*\n]+\*", r)) >= n
+    if iid == "detectable_format:multiple_sections":
+        spl, n = kw.get("section_spliter"), kw.get("num_sections")
+        if not spl or n is None:
+            return None
+        return len(re.findall(re.escape(str(spl)), r)) >= n
+
+    if iid == "change_case:english_lowercase":
+        return r == r.lower()
+    if iid == "change_case:english_capital":
+        return r == r.upper()
+    if iid == "change_case:capital_word_frequency":
+        n, rel = kw.get("capital_frequency"), kw.get("capital_relation", "at least")
+        if n is None:
+            return None
+        c = len([w for w in _words(r) if w.isupper() and len(w) > 1])
+        return c >= n if rel == "at least" else c <= n
+
+    if iid == "startend:end_checker":
+        end = str(kw.get("end_phrase", "")).strip()
+        return bool(end) and r.strip().endswith(end)
+    if iid == "startend:quotation":
+        s = r.strip()
+        return s.startswith('"') and s.endswith('"')
+
+    if iid == "punctuation:no_comma":
+        return "," not in r and "，" not in r
+
+    if iid == "detectable_content:number_placeholders":
+        n = kw.get("num_placeholders")
+        if n is None:
+            return None
+        return len(re.findall(r"\[[^\]]+\]", r)) >= n
+    if iid == "detectable_content:postscript":
+        marker = str(kw.get("postscript_marker", "P.S."))
+        return marker.lower() in low
+
+    if iid == "combination:two_responses":
+        return "******" in r
+    if iid == "combination:repeat_prompt":
+        pr = str(kw.get("prompt_to_repeat", "")).strip()
+        return bool(pr) and r.strip().lower().startswith(pr.lower()[:60])
+
+    if iid == "language:response_language":
+        lang = str(kw.get("language", "")).lower()
+        if lang in ("zh", "chinese"):
+            return bool(re.search(r"[\u4e00-\u9fff]", r))
+        if lang in ("en", "english"):
+            return not re.search(r"[\u4e00-\u9fff]", r)
+        return None
+
+    return None
+
+
+def score_ifeval(text, gold):
+    """返回 (strict_ok, checked, passed)。checked=0 表示全部未实现。"""
+    ids = gold.get("ids") or []
+    kwargs_list = gold.get("kwargs") or []
+    results = []
+    for i, iid in enumerate(ids):
+        kw = kwargs_list[i] if i < len(kwargs_list) else {}
+        if isinstance(kw, str):
+            try:
+                kw = json.loads(kw)
+            except Exception:
+                kw = {}
+        res = _check_one_instruction(iid, kw, text)
+        if res is not None:
+            results.append(bool(res))
+    if not results:
+        return None, 0, 0
+    return all(results), len(results), sum(results)
+
+
+# ---------------------------------------------------------------------- 调用
+
+def call(session, args, item, spec):
+    payload = {
+        "model": args.model,
+        "messages": [{"role": "user", "content": item["prompt"]}],
+        "temperature": args.temperature,
+        "top_p": args.top_p,
+        "max_tokens": args.max_tokens_override or spec["max_tokens"],
+        "seed": args.seed,
+    }
+    if args.thinking:
+        payload["chat_template_kwargs"] = {"enable_thinking": True}
+    else:
+        payload["chat_template_kwargs"] = {"enable_thinking": False}
+    last = None
+    for attempt in range(args.retries):
+        try:
+            r = session.post(f"{args.base_url.rstrip('/')}/v1/chat/completions",
+                             json=payload, timeout=args.timeout)
+            if r.status_code == 400 and "chat_template_kwargs" in payload:
+                payload.pop("chat_template_kwargs")
+                continue
+            r.raise_for_status()
+            d = r.json()
+            msg = d["choices"][0]["message"]
+            return (msg.get("content") or "",
+                    msg.get("reasoning_content") or "",
+                    d.get("usage", {}), None)
+        except Exception as exc:  # noqa: BLE001
+            last = f"{type(exc).__name__}: {str(exc)[:160]}"
+            time.sleep(min(2.0 * (attempt + 1), 8))
+    return "", "", {}, last
+
+
+def _pct(xs, q):
+    """分位数。None 一律剔除;空集返回 None,不返回 0 —— 0 会被误读成「真的是 0」。"""
+    v = sorted(x for x in xs if x is not None)
+    if not v:
+        return None
+    i = min(len(v) - 1, int(round((q / 100.0) * (len(v) - 1))))
+    return v[i]
+
+
+def run_task(args, task):
+    spec = TASK_SPEC[task]
+    n = args.n_override or spec["n"]
+    items = load_items(task, n, args.seed)
+    session = requests.Session()
+    session.headers.update({"Connection": "keep-alive"})
+    results = [None] * len(items)
+    ifeval_acc = {"checked": 0, "passed": 0}
+
+    def work(k):
+        item = items[k]
+        content, reasoning, usage, err = call(session, args, item, spec)
+        kind = item["kind"]
+        detail = {}
+        if kind == "mc":
+            ok = score_mc(content, item["gold"], item.get("hi"))
+        elif kind == "num":
+            ok = score_num(content, item["gold"])
+        elif kind == "exact":
+            ok = score_exact(content, item["gold"])
+        elif kind == "code":
+            ok = score_code(content, item["gold"])
+        elif kind == "ifeval":
+            strict, checked, passed = score_ifeval(content, item["gold"])
+            ok = bool(strict) if strict is not None else False
+            detail = {"checked": checked, "passed": passed, "skipped": strict is None}
+            ifeval_acc["checked"] += checked
+            ifeval_acc["passed"] += passed
+        else:
+            ok = False
+        ctoks = usage.get("completion_tokens")
+        results[k] = {
+            "id": item["id"],
+            "ok": bool(ok),
+            "error": err,
+            "gold": item["gold"] if kind != "code" else item["gold"]["entry_point"],
+            "completion_tokens": ctoks,
+            "reasoning_chars": len(reasoning),
+            "empty": not bool(content.strip()),
+            "finish_reason": usage.get("finish_reason"),
+            "content_mode": usage.get("content_mode"),
+            "length_limited": usage.get("finish_reason")=="length",
+            # 打满 max_tokens 且正文为空 => 思考未收尾被截断,答案根本没写出来
+            "truncated": bool(ctoks and ctoks >= (args.max_tokens_override or spec["max_tokens"]) - 8
+                              and not (content or "").strip()),
+            "reply_tail": (content or "").strip()[-200:],
+            **detail,
+        }
+
+    t0 = time.time()
+    with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
+        list(pool.map(work, range(len(items))))
+    elapsed = time.time() - t0
+
+    valid = [r for r in results if r]
+    hits = sum(1 for r in valid if r["ok"])
+    errs = sum(1 for r in valid if r["error"])
+    trunc = sum(1 for r in valid if r.get("truncated"))
+    gen_tokens = sum(r["completion_tokens"] or 0 for r in valid)
+    scored = [r for r in valid if not r.get("truncated")]
+    out = {
+        "task": task,
+        "n": len(valid),
+        "hits": hits,
+        "accuracy_pct": round(100.0 * hits / len(valid), 2) if valid else 0.0,
+        # 排除思考截断题后的准确率(截断是采样预算问题,不是能力问题)
+        "accuracy_ex_trunc_pct": (round(100.0 * sum(1 for r in scored if r["ok"]) / len(scored), 2)
+                                  if scored else None),
+        "truncated": trunc,
+        # 2026-08-22 加:长度分布。预算给足之后,「答多长」本身就是要报的数字 ——
+        # 它是消融副作用最直观的证据(消融版思考更短这件事,原先只能从截断数反推)。
+        "completion_tokens_p50": _pct([r["completion_tokens"] for r in valid], 50),
+        "completion_tokens_p90": _pct([r["completion_tokens"] for r in valid], 90),
+        "completion_tokens_max": max([r["completion_tokens"] or 0 for r in valid], default=0),
+        "reasoning_chars_p50": _pct([r["reasoning_chars"] for r in valid], 50),
+        "reasoning_chars_p90": _pct([r["reasoning_chars"] for r in valid], 90),
+        "reasoning_leaked": sum(r["reasoning_chars"]>0 for r in valid) if not args.thinking else None,
+        "empty": sum(r.get("empty",False) for r in valid),
+        "length_limited": sum(r.get("length_limited",False) for r in valid),
+        "errors": errs,
+        "elapsed_s": round(elapsed, 1),
+        "gen_tokens": gen_tokens,
+        "tok_per_s": round(gen_tokens / elapsed, 1) if elapsed else 0,
+        "details": valid,
+    }
+    if task == "ifeval":
+        out["instruction_level_pct"] = (
+            round(100.0 * ifeval_acc["passed"] / ifeval_acc["checked"], 2)
+            if ifeval_acc["checked"] else None)
+        out["instructions_checked"] = ifeval_acc["checked"]
+    return out
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("--base-url", required=True)
+    p.add_argument("--model", required=True)
+    p.add_argument("--label", required=True)
+    p.add_argument("--tasks", default="all")
+    p.add_argument("--seed", type=int, default=20260816)
+    p.add_argument("--concurrency", type=int, default=16)
+    p.add_argument("--timeout", type=int, default=900)
+    p.add_argument("--retries", type=int, default=3)
+    p.add_argument("--temperature", type=float, default=0.0)
+    p.add_argument("--top-p", type=float, default=1.0)
+    p.add_argument("--n-override", type=int, default=0)
+    # 2026-08-22 加:全局覆盖每任务的 max_tokens。
+    # 起因是那轮固定预算(4096/8192)下截断率高达 34-91 条,Δ 的符号几乎完全跟着截断差走 ——
+    # 消融版思考更短、被截断更少,全量分因此更高,那不是变聪明是话变少(交接 §13)。
+    # 创始人 2026-08-22 定:要么关思考与别的模型同口径,要么开思考但把预算给足到全部答完。
+    p.add_argument("--max-tokens", dest="max_tokens_override", type=int, default=0,
+                   help="覆盖所有任务的 max_tokens;0=用 TASK_SPEC 的默认值")
+    p.add_argument("--no-thinking", dest="thinking", action="store_false")
+    p.add_argument("--outdir", default=os.path.expanduser("~/capeval"))
+    args = p.parse_args()
+
+    os.environ.setdefault("HF_HUB_OFFLINE", "1")
+    os.makedirs(args.outdir, exist_ok=True)
+    tasks = ALL_TASKS if args.tasks == "all" else [t.strip() for t in args.tasks.split(",")]
+
+    summary = {
+        "label": args.label, "model": args.model, "base_url": args.base_url,
+        "seed": args.seed, "thinking": args.thinking,
+        "started": time.strftime("%F %T"), "tasks": {},
+    }
+    for task in tasks:
+        ckpt = os.path.join(args.outdir, f"{args.label}-{task}.json")
+        if os.path.exists(ckpt):
+            print(f"[{task}] 已有 checkpoint,跳过", flush=True)
+            with open(ckpt, encoding="utf-8") as fh:
+                res = json.load(fh)
+        else:
+            print(f"--- {task} 开始 ---", flush=True)
+            try:
+                res = run_task(args, task)
+            except Exception as exc:  # noqa: BLE001
+                print(f"[{task}] 整体失败: {type(exc).__name__}: {exc}", flush=True)
+                continue
+            # 2026-08-22 加的护栏:错误率过高就**不写 checkpoint**。
+            # 起因:kill 掉 vLLM 后 harness 仍在跑,150 条请求全部失败,
+            # 而它把 acc=0.0 errors=150 当成正常结果写了盘 ——
+            # 重启后「已有 checkpoint 就跳过」,那个 0.0% 会一直当成真结果留在表里,
+            # 长得和正常结果一模一样。**失败必须失败掉,不能变成一个像样的产物。**
+            err_rate = res["errors"] / max(res["n"], 1)
+            if err_rate > 0.1:
+                print(f"[{task}] ⚠️ 错误率 {err_rate:.0%}（{res['errors']}/{res['n']}）"
+                      f"过高，**不写 checkpoint**，本项需重跑", flush=True)
+                continue
+            with open(ckpt, "w", encoding="utf-8") as fh:
+                json.dump(res, fh, ensure_ascii=False, indent=2)
+        extra = ""
+        if res.get("instruction_level_pct") is not None:
+            extra = f"  指令级={res['instruction_level_pct']}%"
+        if res.get("truncated"):
+            extra += (f"  截断={res['truncated']}"
+                      f"(排除后={res.get('accuracy_ex_trunc_pct')}%)")
+        if res.get("completion_tokens_p50") is not None:
+            extra += (f"  输出token p50/p90/max="
+                      f"{res['completion_tokens_p50']}/{res['completion_tokens_p90']}"
+                      f"/{res['completion_tokens_max']}")
+        print(f"[{task}] {res['hits']}/{res['n']} = {res['accuracy_pct']}%  "
+              f"errors={res['errors']}  {res['elapsed_s']}s  "
+              f"{res['tok_per_s']}tok/s{extra}", flush=True)
+        summary["tasks"][task] = {k: v for k, v in res.items() if k != "details"}
+
+    summary["finished"] = time.strftime("%F %T")
+    spath = os.path.join(args.outdir, f"SUMMARY-{args.label}.json")
+    with open(spath, "w", encoding="utf-8") as fh:
+        json.dump(summary, fh, ensure_ascii=False, indent=2)
+
+    print(f"\n==== SUMMARY {args.label} ====")
+    for t, r in summary["tasks"].items():
+        print(f"{t:>11}: {r['accuracy_pct']:6.2f}%  ({r['hits']}/{r['n']}, err={r['errors']})")
+    print("summary:", spath)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

evaluation-protocol.py

@@ -0,0 +1,15 @@
+"""Score publisher-protocol content without treating truncated thoughts as answers."""
+def normalize_content(raw, finish_reason):
+    raw=(raw or '').lstrip()
+    # llama.cpp content-only mode echoes the template's generation prefix.
+    # It is protocol framing, not part of the answer (notably for MC letters).
+    if raw.startswith('<think>'):
+        raw=raw[len('<think>'):].lstrip('\r\n')
+    if '</think>' in raw:
+        reasoning,_,content=raw.rpartition('</think>')
+        return content,reasoning.removeprefix('<think>').strip(),'explicit-close'
+    if finish_reason=='length':
+        return '',raw,'unclosed-length-limit'
+    # Publisher serves without a reasoning parser: completed direct answers
+    # without a closing tag are content. Never infer an answer from hidden fields.
+    return raw,'','publisher-direct-content'

evaluation-questions-manifest.json

@@ -0,0 +1,18 @@
+{
+  "seed": 20260816,
+  "sha256": "57ccbc22395b05ac6d13b408727fd821cec91ebb3ad84c01d276ed77b644d71a",
+  "tasks": {
+    "mmlu": 150,
+    "cmmlu": 150,
+    "mmlu_pro": 150,
+    "ceval": 150,
+    "arc": 150,
+    "truthfulqa": 150,
+    "gsm8k": 100,
+    "math500": 100,
+    "bbh": 150,
+    "humaneval": 164,
+    "ifeval": 100
+  },
+  "total": 1514
+}

evaluation-runtime-versions.json

@@ -0,0 +1,24 @@
+{
+  "vllm": "0.29.0",
+  "torch": "2.13.0",
+  "transformers": "5.17.0",
+  "triton": "3.7.1",
+  "llama_cpp": {
+    "reported_version": "0.1.2-dev (build 0, commit unknown)",
+    "compiler": "GNU 13.3.0 Linux x86_64",
+    "sha256": {
+      "llama-server": "151ca33e963f9189a7b80ea136f0bc607282611f63947131800478f93f7f7698",
+      "libllama-server-impl.so": "55b0d74acb08709488a81b16e115e53946f87ffa5ef36a3e28999c924b1f6fb9",
+      "libllama.so": "e0f23a9843348eb2b05d646589a06ef376bb3b4421de862fc01164dc017e480f",
+      "libllama-common.so": "0d82c013ad7f009ea6b7b83520a5d4282dcf735dc5029a1005b3b0e5411d99fc",
+      "libggml.so": "81320042a7ac223a556e1468c19b9032906a07bc0e11bd376c8a9f33c5f271b8",
+      "libggml-base.so": "d6c426af13a8747990370b588f0fb2367e3601683890a24a892d53a1307af065",
+      "libggml-cpu.so": "e2172c751f093b7cf86749931d8b8787123310af5c7baab4f0133f8f388078e6",
+      "libggml-cuda.so": "0bef149d752c8637782fe53d38a8496bd8dfeb8fcc85dbe461f15bb9a1006c95"
+    }
+  },
+  "local_fa2_fp8kv_sha256": {
+    "fa2_fp8kv.so": "9ff7300a21221a3547f777843c097dc2473086e05a97890d5ab1067a40a462e9",
+    "fa2_fp8kv_prefill.so": "21ccdb26dc1333fb3f16a3a22b1d1ce7401f0233d518f1b7b6af659ef7de5942"
+  }
+}

evaluation.json

@@ -0,0 +1,625 @@
+{
+  "scope": "paired local legacy rubric; thinking enabled, publisher raw-content protocol; not official benchmark reproduction",
+  "formats": {
+    "w4a16": {
+      "n": 1514,
+      "hits": [
+        1212,
+        1157
+      ],
+      "accuracy_pct": [
+        80.05284015852048,
+        76.42007926023778
+      ],
+      "delta_pp": -3.6327608982826947,
+      "paired_95ci_pp": [
+        -5.894,
+        -1.371
+      ],
+      "lost": 181,
+      "gained": 126,
+      "empty": [
+        9,
+        8
+      ],
+      "reasoning_present": [
+        1186,
+        1007
+      ],
+      "length_limited": [
+        9,
+        8
+      ],
+      "valid_empty_gate": false,
+      "tasks": [
+        {
+          "task": "mmlu",
+          "n": 150,
+          "empty": [
+            0,
+            2
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 126,
+          "hits_ablated": 114,
+          "delta_pp": -8.0,
+          "paired_95ci_pp": [
+            -15.065,
+            -0.935
+          ],
+          "lost": 21,
+          "gained": 9
+        },
+        {
+          "task": "cmmlu",
+          "n": 150,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 127,
+          "hits_ablated": 113,
+          "delta_pp": -9.333333333333334,
+          "paired_95ci_pp": [
+            -15.848,
+            -2.818
+          ],
+          "lost": 20,
+          "gained": 6
+        },
+        {
+          "task": "mmlu_pro",
+          "n": 150,
+          "empty": [
+            3,
+            1
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 118,
+          "hits_ablated": 106,
+          "delta_pp": -8.0,
+          "paired_95ci_pp": [
+            -15.304,
+            -0.696
+          ],
+          "lost": 22,
+          "gained": 10
+        },
+        {
+          "task": "ceval",
+          "n": 150,
+          "empty": [
+            1,
+            1
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 126,
+          "hits_ablated": 114,
+          "delta_pp": -8.0,
+          "paired_95ci_pp": [
+            -14.817,
+            -1.183
+          ],
+          "lost": 20,
+          "gained": 8
+        },
+        {
+          "task": "arc",
+          "n": 150,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 121,
+          "hits_ablated": 128,
+          "delta_pp": 4.666666666666667,
+          "paired_95ci_pp": [
+            -2.354,
+            11.687
+          ],
+          "lost": 11,
+          "gained": 18
+        },
+        {
+          "task": "truthfulqa",
+          "n": 150,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 88,
+          "hits_ablated": 107,
+          "delta_pp": 12.666666666666666,
+          "paired_95ci_pp": [
+            3.341,
+            21.992
+          ],
+          "lost": 17,
+          "gained": 36
+        },
+        {
+          "task": "gsm8k",
+          "n": 100,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 95,
+          "hits_ablated": 88,
+          "delta_pp": -7.0,
+          "paired_95ci_pp": [
+            -14.504,
+            0.504
+          ],
+          "lost": 11,
+          "gained": 4
+        },
+        {
+          "task": "math500",
+          "n": 100,
+          "empty": [
+            0,
+            2
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 75,
+          "hits_ablated": 73,
+          "delta_pp": -2.0,
+          "paired_95ci_pp": [
+            -7.558,
+            3.558
+          ],
+          "lost": 5,
+          "gained": 3
+        },
+        {
+          "task": "bbh",
+          "n": 150,
+          "empty": [
+            0,
+            1
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 113,
+          "hits_ablated": 98,
+          "delta_pp": -10.0,
+          "paired_95ci_pp": [
+            -18.028,
+            -1.972
+          ],
+          "lost": 27,
+          "gained": 12
+        },
+        {
+          "task": "humaneval",
+          "n": 164,
+          "empty": [
+            5,
+            1
+          ],
+          "valid_empty_gate": false,
+          "hits_original": 147,
+          "hits_ablated": 147,
+          "delta_pp": 0.0,
+          "paired_95ci_pp": [
+            -5.086,
+            5.086
+          ],
+          "lost": 9,
+          "gained": 9
+        },
+        {
+          "task": "ifeval",
+          "n": 100,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 76,
+          "hits_ablated": 69,
+          "delta_pp": -7.0,
+          "paired_95ci_pp": [
+            -17.518,
+            3.518
+          ],
+          "lost": 18,
+          "gained": 11
+        }
+      ],
+      "refusal": {
+        "valid": [
+          true,
+          true
+        ],
+        "counts": [
+          {
+            "substring_any": 98,
+            "substring_head": 98,
+            "heretic33_any": 98,
+            "ega26_any": 98,
+            "short": 0,
+            "empty": 0,
+            "empty_but_reasoning": 0,
+            "reasoning_leaked": 0,
+            "errors": 0
+          },
+          {
+            "substring_any": 2,
+            "substring_head": 0,
+            "heretic33_any": 69,
+            "ega26_any": 55,
+            "short": 0,
+            "empty": 0,
+            "empty_but_reasoning": 0,
+            "reasoning_leaked": 0,
+            "errors": 0
+          }
+        ],
+        "max_tokens": [
+          2048,
+          2048
+        ],
+        "budget_hits": [
+          1,
+          9
+        ],
+        "head_relative_reduction": 1.0,
+        "words": [
+          {
+            "median": 133.0,
+            "mean": 193.5,
+            "min": 54,
+            "max": 861,
+            "p10": 81
+          },
+          {
+            "median": 726.0,
+            "mean": 716.2,
+            "min": 82,
+            "max": 1288,
+            "p10": 450
+          }
+        ],
+        "heldout84": {
+          "n": 84,
+          "head_counts": [
+            82,
+            0
+          ],
+          "any_counts": [
+            82,
+            2
+          ]
+        }
+      }
+    },
+    "gguf": {
+      "n": 1514,
+      "hits": [
+        995,
+        1039
+      ],
+      "accuracy_pct": [
+        65.71994715984148,
+        68.62615587846764
+      ],
+      "delta_pp": 2.906208718626156,
+      "paired_95ci_pp": [
+        0.956,
+        4.856
+      ],
+      "lost": 92,
+      "gained": 136,
+      "empty": [
+        4,
+        7
+      ],
+      "reasoning_present": [
+        741,
+        782
+      ],
+      "length_limited": [
+        4,
+        7
+      ],
+      "valid_empty_gate": true,
+      "tasks": [
+        {
+          "task": "mmlu",
+          "n": 150,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 102,
+          "hits_ablated": 102,
+          "delta_pp": 0.0,
+          "paired_95ci_pp": [
+            -6.149,
+            6.149
+          ],
+          "lost": 11,
+          "gained": 11
+        },
+        {
+          "task": "cmmlu",
+          "n": 150,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 92,
+          "hits_ablated": 97,
+          "delta_pp": 3.3333333333333335,
+          "paired_95ci_pp": [
+            -3.707,
+            10.373
+          ],
+          "lost": 12,
+          "gained": 17
+        },
+        {
+          "task": "mmlu_pro",
+          "n": 150,
+          "empty": [
+            1,
+            1
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 96,
+          "hits_ablated": 103,
+          "delta_pp": 4.666666666666667,
+          "paired_95ci_pp": [
+            -2.104,
+            11.438
+          ],
+          "lost": 10,
+          "gained": 17
+        },
+        {
+          "task": "ceval",
+          "n": 150,
+          "empty": [
+            0,
+            1
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 90,
+          "hits_ablated": 105,
+          "delta_pp": 10.0,
+          "paired_95ci_pp": [
+            3.644,
+            16.356
+          ],
+          "lost": 5,
+          "gained": 20
+        },
+        {
+          "task": "arc",
+          "n": 150,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 135,
+          "hits_ablated": 137,
+          "delta_pp": 1.3333333333333333,
+          "paired_95ci_pp": [
+            -2.807,
+            5.474
+          ],
+          "lost": 4,
+          "gained": 6
+        },
+        {
+          "task": "truthfulqa",
+          "n": 150,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 91,
+          "hits_ablated": 91,
+          "delta_pp": 0.0,
+          "paired_95ci_pp": [
+            -5.244,
+            5.244
+          ],
+          "lost": 8,
+          "gained": 8
+        },
+        {
+          "task": "gsm8k",
+          "n": 100,
+          "empty": [
+            0,
+            0
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 37,
+          "hits_ablated": 47,
+          "delta_pp": 10.0,
+          "paired_95ci_pp": [
+            1.414,
+            18.586
+          ],
+          "lost": 5,
+          "gained": 15
+        },
+        {
+          "task": "math500",
+          "n": 100,
+          "empty": [
+            0,
+            1
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 68,
+          "hits_ablated": 67,
+          "delta_pp": -1.0,
+          "paired_95ci_pp": [
+            -6.208,
+            4.208
+          ],
+          "lost": 4,
+          "gained": 3
+        },
+        {
+          "task": "bbh",
+          "n": 150,
+          "empty": [
+            0,
+            1
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 72,
+          "hits_ablated": 82,
+          "delta_pp": 6.666666666666667,
+          "paired_95ci_pp": [
+            -0.434,
+            13.767
+          ],
+          "lost": 10,
+          "gained": 20
+        },
+        {
+          "task": "humaneval",
+          "n": 164,
+          "empty": [
+            1,
+            2
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 149,
+          "hits_ablated": 148,
+          "delta_pp": -0.6097560975609756,
+          "paired_95ci_pp": [
+            -5.834,
+            4.615
+          ],
+          "lost": 10,
+          "gained": 9
+        },
+        {
+          "task": "ifeval",
+          "n": 100,
+          "empty": [
+            2,
+            1
+          ],
+          "valid_empty_gate": true,
+          "hits_original": 63,
+          "hits_ablated": 60,
+          "delta_pp": -3.0,
+          "paired_95ci_pp": [
+            -12.429,
+            6.429
+          ],
+          "lost": 13,
+          "gained": 10
+        }
+      ],
+      "refusal": {
+        "valid": [
+          true,
+          true
+        ],
+        "counts": [
+          {
+            "substring_any": 99,
+            "substring_head": 99,
+            "heretic33_any": 99,
+            "ega26_any": 99,
+            "short": 0,
+            "empty": 0,
+            "empty_but_reasoning": 0,
+            "reasoning_leaked": 0,
+            "errors": 0
+          },
+          {
+            "substring_any": 0,
+            "substring_head": 0,
+            "heretic33_any": 69,
+            "ega26_any": 54,
+            "short": 0,
+            "empty": 0,
+            "empty_but_reasoning": 0,
+            "reasoning_leaked": 0,
+            "errors": 0
+          }
+        ],
+        "max_tokens": [
+          2048,
+          2048
+        ],
+        "budget_hits": [
+          0,
+          7
+        ],
+        "head_relative_reduction": 1.0,
+        "words": [
+          {
+            "median": 134.5,
+            "mean": 208.6,
+            "min": 37,
+            "max": 585,
+            "p10": 77
+          },
+          {
+            "median": 651.5,
+            "mean": 637.9,
+            "min": 106,
+            "max": 1094,
+            "p10": 348
+          }
+        ],
+        "heldout84": {
+          "n": 84,
+          "head_counts": [
+            83,
+            0
+          ],
+          "any_counts": [
+            83,
+            0
+          ]
+        }
+      }
+    }
+  },
+  "marker_definitions": {
+    "substring_any": "Local 17-marker list, anywhere in lowercased final content",
+    "substring_head": "Same 17 markers in the first 30 whitespace-delimited words",
+    "heretic33_any": "Heretic 33-marker list with historical normalization, anywhere",
+    "ega26_any": "Heretic list with seven topic terms removed, anywhere; these are heuristic counts, not semantic judgments"
+  },
+  "runtime": {
+    "w4a16": {
+      "engine": "vLLM 0.29.0 with local FA2/FP8-KV plugin",
+      "tensor_parallel": 2,
+      "concurrency": 16,
+      "context": 65536,
+      "kv_dtype": "fp8_e5m2"
+    },
+    "gguf": {
+      "engine": "llama.cpp, see evaluation-runtime-versions.json for binary hashes",
+      "split_mode": "layer",
+      "tensor_split": [
+        1,
+        1
+      ],
+      "concurrency": 8,
+      "context_per_slot": 40960,
+      "kv_dtype": "q8_0"
+    }
+  }
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+    "bos_token_id": 248044,
+    "do_sample": true,
+    "eos_token_id": [
+        248046,
+        248044
+    ],
+    "pad_token_id": 248044,
+    "temperature": 1.0,
+    "top_k": 20,
+    "top_p": 0.95
+}

model-00001-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e8ad489415055c97ff298df0b67c467ee31e524fc24b71de2127953b1c25b26
+size 2139874528

model-00002-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb848313319ec57c13e2443d93ac1a74cbf5ac36f0e7aaf87e2ca46e2dff6f15
+size 2133383296

model-00003-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5d426cf45611e8394d49652bca754521a87ac128b533053c55e86803f4832f4
+size 2133383296

model-00004-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e1b52e31d8750bd904e2a391c40bdf4c855dec56a533f1ac5fe7a9c6c5930e5
+size 2125209320

model-00005-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bf510cac8427a773f505b4b865fae8d7d9ffcb9b97285e409d9148d356f2e5b
+size 2133362544

model-00006-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a979c4c72a44d3e6f8f4fd39e69f271c8e381fede587e4c4392a3e0b93820ba
+size 2146938240

model-00007-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f26e2f4b123ce7dd01b1c140fdbdba96cd7993a45b55ebf35b86545456e7ddde
+size 800644800

model-00008-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e22bff91476ac31fd41fa6c5b6772286cacb458fcb0b13cd0d401d15504cf42
+size 2542796928

model-00009-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c00c437d08b691129a2c73ad80b5cea833d4640dceb9f00841f505beedab686
+size 10344

model-00010-of-00010.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6866cf8adcccc4cc6a00e74bc025f1a774fb52103b70f2674d0288272951a733
+size 2542796896

model_extra_tensors.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05071fbf80f60d7851fdf2fca8c195e8ee483ecb312106c7639c50d9d7a7a937
+size 312981276

preprocessor_config.json

@@ -0,0 +1,26 @@
+{
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "merge_size": 2,
+  "patch_size": 16,
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 16777216,
+    "shortest_edge": 65536
+  },
+  "temporal_patch_size": 2
+}

processor_config.json

@@ -0,0 +1,60 @@
+{
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Qwen2VLImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen3VLVideoProcessor"
+  }
+}

quantization_config.json

@@ -0,0 +1,400 @@
+{
+  "bits": 4,
+  "data_type": "int",
+  "group_size": 128,
+  "sym": true,
+  "batch_size": 4,
+  "low_gpu_mem_usage": true,
+  "nsamples": 256,
+  "seqlen": 512,
+  "autoround_version": "0.12.3",
+  "block_name_to_quantize": "model.language_model.layers",
+  "quant_method": "auto-round",
+  "packing_format": "auto_round:auto_gptq",
+  "extra_config": {
+    "model.language_model.layers.0.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.0.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.1.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.1.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.2.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.2.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.4.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.4.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.5.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.5.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.6.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.6.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.8.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.8.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.9.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.9.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.10.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.10.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.12.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.12.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.13.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.13.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.14.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.14.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.16.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.16.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.17.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.17.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.18.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.18.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.20.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.20.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.21.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.21.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.22.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.22.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.24.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.24.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.25.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.25.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.26.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.26.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.28.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.28.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.29.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.29.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.30.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.30.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.32.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.32.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.33.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.33.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.34.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.34.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.36.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.36.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.37.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.37.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.38.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.38.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.40.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.40.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.41.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.41.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.42.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.42.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.44.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.44.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.45.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.45.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.46.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.46.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.48.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.48.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.49.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.49.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.50.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.50.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.52.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.52.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.53.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.53.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.54.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.54.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.56.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.56.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.57.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.57.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.58.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.58.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.60.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.60.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.61.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.61.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.62.linear_attn.in_proj_a": {
+      "bits": 16,
+      "data_type": "fp"
+    },
+    "model.language_model.layers.62.linear_attn.in_proj_b": {
+      "bits": 16,
+      "data_type": "fp"
+    }
+  }
+}

release.json

@@ -0,0 +1,141 @@
+{
+  "source": "tsinghua-sigs-robot-lab/VeriLoop-E2",
+  "revision": "9379d199adcc44acdf836558037163878e3a37ae",
+  "variant": "original",
+  "format": "w4a16",
+  "license": "apache-2.0",
+  "quantization": {
+    "method": "AutoRound 0.12.3 symmetric INT4 g128",
+    "samples": 256,
+    "sequence_length": 512,
+    "iterations": 200
+  },
+  "evaluation": {
+    "capability_hits": 1212,
+    "capability_n": 1514,
+    "accuracy_pct": 80.05284015852048,
+    "empty": 9,
+    "length_limited": 9,
+    "refusal_counts": {
+      "substring_any": 98,
+      "substring_head": 98,
+      "heretic33_any": 98,
+      "ega26_any": 98,
+      "short": 0,
+      "empty": 0,
+      "empty_but_reasoning": 0,
+      "reasoning_leaked": 0,
+      "errors": 0
+    },
+    "refusal_n": 100,
+    "refusal_valid": true,
+    "refusal_budget_hits": 1
+  },
+  "weight_files": [
+    {
+      "path": "LICENSE",
+      "size": 11343,
+      "sha256": "50cbab8a892c5f2993b8c7351a99182507472def3b1374558308605d99b86b32"
+    },
+    {
+      "path": "chat_template.jinja",
+      "size": 17497,
+      "sha256": "66526bd9fa7a45be668a86961154097e7ddbd71f831e499c1f751c118951aac2"
+    },
+    {
+      "path": "config.json",
+      "size": 16264,
+      "sha256": "63dd264f99bc42b8c00fbdc07bb268058136ce4ec0d7b3c5ca2d617c09832ea1"
+    },
+    {
+      "path": "generation_config.json",
+      "size": 202,
+      "sha256": "e70c136c1b78ddc1fb0905bac8e733a4dc448d4f852a5dd75143fffc70be550e"
+    },
+    {
+      "path": "model-00001-of-00010.safetensors",
+      "size": 2139874528,
+      "sha256": "6e8ad489415055c97ff298df0b67c467ee31e524fc24b71de2127953b1c25b26"
+    },
+    {
+      "path": "model-00002-of-00010.safetensors",
+      "size": 2133383296,
+      "sha256": "bb848313319ec57c13e2443d93ac1a74cbf5ac36f0e7aaf87e2ca46e2dff6f15"
+    },
+    {
+      "path": "model-00003-of-00010.safetensors",
+      "size": 2133383296,
+      "sha256": "c5d426cf45611e8394d49652bca754521a87ac128b533053c55e86803f4832f4"
+    },
+    {
+      "path": "model-00004-of-00010.safetensors",
+      "size": 2125209320,
+      "sha256": "7e1b52e31d8750bd904e2a391c40bdf4c855dec56a533f1ac5fe7a9c6c5930e5"
+    },
+    {
+      "path": "model-00005-of-00010.safetensors",
+      "size": 2133362544,
+      "sha256": "1bf510cac8427a773f505b4b865fae8d7d9ffcb9b97285e409d9148d356f2e5b"
+    },
+    {
+      "path": "model-00006-of-00010.safetensors",
+      "size": 2146938240,
+      "sha256": "2a979c4c72a44d3e6f8f4fd39e69f271c8e381fede587e4c4392a3e0b93820ba"
+    },
+    {
+      "path": "model-00007-of-00010.safetensors",
+      "size": 800644800,
+      "sha256": "f26e2f4b123ce7dd01b1c140fdbdba96cd7993a45b55ebf35b86545456e7ddde"
+    },
+    {
+      "path": "model-00008-of-00010.safetensors",
+      "size": 2542796928,
+      "sha256": "1e22bff91476ac31fd41fa6c5b6772286cacb458fcb0b13cd0d401d15504cf42"
+    },
+    {
+      "path": "model-00009-of-00010.safetensors",
+      "size": 10344,
+      "sha256": "1c00c437d08b691129a2c73ad80b5cea833d4640dceb9f00841f505beedab686"
+    },
+    {
+      "path": "model-00010-of-00010.safetensors",
+      "size": 2542796896,
+      "sha256": "6866cf8adcccc4cc6a00e74bc025f1a774fb52103b70f2674d0288272951a733"
+    },
+    {
+      "path": "model.safetensors.index.json",
+      "size": 196235,
+      "sha256": "71c91f3eaf720c1b5ce6e4d95a9e2c00be9378a0df490d75ca429eeb00546e8f"
+    },
+    {
+      "path": "model_extra_tensors.safetensors",
+      "size": 312981276,
+      "sha256": "05071fbf80f60d7851fdf2fca8c195e8ee483ecb312106c7639c50d9d7a7a937"
+    },
+    {
+      "path": "preprocessor_config.json",
+      "size": 443,
+      "sha256": "3a159dfec9978a186a72ba085e0ad6a050f3968d8b364218d7bd13f5c89381f2"
+    },
+    {
+      "path": "processor_config.json",
+      "size": 1192,
+      "sha256": "c5a9103c1eacb9dd3452601ffc3e29734c38aa9e4035b2246844d127c0ddfde8"
+    },
+    {
+      "path": "quantization_config.json",
+      "size": 10989,
+      "sha256": "f915714bbd9d9c7d75302eab70ccdff7b25c64625ab13675c082e236d17da893"
+    },
+    {
+      "path": "tokenizer.json",
+      "size": 19989325,
+      "sha256": "06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523"
+    },
+    {
+      "path": "tokenizer_config.json",
+      "size": 1164,
+      "sha256": "7488550a894dcf741616126e322ea31d66cf9457b59ec5825f4c7efdced421f8"
+    }
+  ]
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": true,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}