Document official ParseBench validation for nano-v2

README.md

@@ -27,7 +27,18 @@ A 1.2B-parameter open-weight document parsing model, packaged and orchestrated b
 
 ## ParseBench results
 
-Official end-to-end validation of these v2 weights is pending. Benchmark results from the original model are not presented as v2 results.
+Measured 2026-10-28 with the official ParseBench harness, **full set, single end-to-end pass** (2,078 unique documents; 2,553 group test cases; 0 inference failures):
+
+| Dimension | Metric | Score |
+|---|---|---|
+| **Overall (mean)** | mean of 5 dimensions | 78.96 |
+| Tables | grits_trm_composite | 86.06 |
+| Visual Grounding / Layout | layout_element_rule_pass_rate | 75.31 |
+| Content Faithfulness | content_faithfulness | 87.64 |
+| Semantic Formatting | semantic_formatting | 76.91 |
+| Charts | rule_pass_rate | 68.87 |
+
+Author-measured with unmodified upstream `7de04fc11c6a692dfc5e1a86c7be817f3f0af8de` and dataset revision `2805a1d940f95a203e0ae4b88be9934f7765b3fc`. All 2,553 officially eligible group cases were scored successfully; the reports additionally retain 32 skipped records outside those case counts. This is one pass, not a repeated-run mean; maintainer reproduction and official leaderboard acceptance are pending. [Full validation summary](evaluation/parsebench-20261028-gpu5.json) · [Reproduction instructions](evaluation/reproduce.md).
 
 ## Serving
 

evaluation/parsebench-20261028-gpu5.json

@@ -0,0 +1,116 @@
+{
+  "schema": "parsebench-author-validation/v1",
+  "model_id": "KDLAI/KDL-Frontier-Parser-nano-v2",
+  "model_revision": "766505d578be5d99f2358ec9c95224023cd28d5f",
+  "model_safetensors_sha256": "596389309a85a67341cf5e6b062aefc1042116314ed365b2eb3ffb2d57924047",
+  "benchmark_repository": "run-llama/ParseBench",
+  "benchmark_revision": "7de04fc11c6a692dfc5e1a86c7be817f3f0af8de",
+  "dataset_id": "llamaindex/ParseBench",
+  "dataset_revision": "2805a1d940f95a203e0ae4b88be9934f7765b3fc",
+  "provider": "kdl_frontier_nano",
+  "replicates": 1,
+  "author_measured": true,
+  "maintainer_reproduction": "not_yet_observed",
+  "unique_documents": 2078,
+  "group_test_cases": 2553,
+  "evaluation_skipped_records": 32,
+  "inference_failures": 0,
+  "inference_skipped": 0,
+  "group_results": {
+    "table": {
+      "metric": "grits_trm_composite",
+      "raw_value": 0.8606003056212997,
+      "score": 86.06003056212998,
+      "coverage": {
+        "total_examples": 503,
+        "successful": 503,
+        "failed": 0,
+        "skipped": 0
+      },
+      "report_sha256": "f51c795aaa21f4d80acaffddbd0073013ea59db160ede3b502851132c58c6968",
+      "started_at": "2026-10-28T12:19:20.974057",
+      "completed_at": "2026-10-28T12:19:21.003449"
+    },
+    "chart": {
+      "metric": "rule_pass_rate",
+      "raw_value": 0.6887017813954434,
+      "score": 68.87017813954434,
+      "coverage": {
+        "total_examples": 568,
+        "successful": 568,
+        "failed": 0,
+        "skipped": 0
+      },
+      "report_sha256": "390f082bee239c6f5847d1c3d7a76d80c1192018c443f3f5558b1eb7c0e8176c",
+      "started_at": "2026-10-28T12:18:33.777658",
+      "completed_at": "2026-10-28T12:18:33.812787"
+    },
+    "text_content": {
+      "metric": "content_faithfulness",
+      "raw_value": 0.876431835862896,
+      "score": 87.6431835862896,
+      "coverage": {
+        "total_examples": 506,
+        "successful": 506,
+        "failed": 0,
+        "skipped": 1
+      },
+      "report_sha256": "0623e8c2f33928753d7e7441155e2c86f63eeba4bf1d18e6f37f8f4a5f27dc3e",
+      "started_at": "2026-10-28T12:20:54.215579",
+      "completed_at": "2026-10-28T12:20:54.365057"
+    },
+    "text_formatting": {
+      "metric": "semantic_formatting",
+      "raw_value": 0.7691130071725973,
+      "score": 76.91130071725973,
+      "coverage": {
+        "total_examples": 476,
+        "successful": 476,
+        "failed": 0,
+        "skipped": 31
+      },
+      "report_sha256": "fcdd64bc31ce727483a291cfa25af322b227f9f228a2947f23b0ade0032cdc0a",
+      "started_at": "2026-10-28T12:21:03.998469",
+      "completed_at": "2026-10-28T12:21:04.052873"
+    },
+    "layout": {
+      "metric": "layout_element_rule_pass_rate",
+      "raw_value": 0.7531206769668802,
+      "score": 75.31206769668802,
+      "coverage": {
+        "total_examples": 500,
+        "successful": 500,
+        "failed": 0,
+        "skipped": 0
+      },
+      "report_sha256": "11128ae4266152c4b3630540fe9c4cd803d636303fbae8711856b47a1e1f541d",
+      "started_at": "2026-10-28T12:18:50.184160",
+      "completed_at": "2026-10-28T12:18:50.255268"
+    }
+  },
+  "scores": {
+    "table": 86.06003056212998,
+    "chart": 68.87017813954434,
+    "text_content": 87.6431835862896,
+    "text_formatting": 76.91130071725973,
+    "layout": 75.31206769668802,
+    "overall": 78.95935214038234
+  },
+  "wall_seconds_including_server_start_and_cleanup": 1396.8262436389923,
+  "runtime": {
+    "image_id": "sha256:609a5b463503f663bedc6a8045ef2aae530bf59c6dc20371c84b1d302fb6f9ca",
+    "image_reference": "vllm/vllm-openai@sha256:61fc8a896b0a4fbbbdc063bc4b0dbc25ce98e02b5050c24aeb7830ac02039b14",
+    "vllm_version": "0.28.0",
+    "hardware": "single NVIDIA B200 GPU",
+    "weight_dtype": "bfloat16",
+    "max_model_len": 8192,
+    "max_num_seqs": 24,
+    "max_concurrent_documents": 20,
+    "per_document_concurrency": 8,
+    "gpu_memory_utilization": "vLLM default; no override",
+    "LLM_normalization": "off",
+    "custom_provider_or_post_normalize_rules": false
+  },
+  "date": "2026-10-28",
+  "scope": "Full-set single end-to-end pass using the unchanged upstream multi-stage layout/region provider; not a one-request-per-page model or a repeated-run mean."
+}

evaluation/reproduce.md

@@ -0,0 +1,46 @@
+# Reproduce the published nano-v2 validation
+
+Model: `KDLAI/KDL-Frontier-Parser-nano-v2@766505d578be5d99f2358ec9c95224023cd28d5f`. Use the original upstream `kdl_frontier_nano` provider unchanged. A full end-to-end pass uses layout detection and multiple region-recognition calls; it does not mean a single full-page OCR request.
+
+## Pinned source and data
+
+```bash
+git clone https://github.com/run-llama/ParseBench.git
+cd ParseBench
+git checkout --detach 7de04fc11c6a692dfc5e1a86c7be817f3f0af8de
+uv sync --frozen --python /usr/bin/python3.12 --extra openai
+.venv/bin/python -c "from huggingface_hub import snapshot_download; snapshot_download('KDLAI/KDL-Frontier-Parser-nano-v2', revision='766505d578be5d99f2358ec9c95224023cd28d5f', local_dir='model-v2', token=False); snapshot_download('llamaindex/ParseBench', repo_type='dataset', revision='2805a1d940f95a203e0ae4b88be9934f7765b3fc', local_dir='data', token=False)"
+```
+
+## Serve the pinned downloaded weights
+
+The validation used one NVIDIA B200, vLLM 0.28.0, BF16, no GPU-memory-utilization override, local image ID `sha256:609a5b463503f663bedc6a8045ef2aae530bf59c6dc20371c84b1d302fb6f9ca` and registry-pinned image `vllm/vllm-openai@sha256:61fc8a896b0a4fbbbdc063bc4b0dbc25ce98e02b5050c24aeb7830ac02039b14`. Set `GPU_UUID` to the device assigned to your run. Use a fresh container name and output directory.
+
+```bash
+export GPU_UUID='<your assigned GPU UUID>'
+docker run --rm --name parsebench-nano-v2-reproduction \
+  --gpus "device=$GPU_UUID" --shm-size 8g \
+  -p 127.0.0.1:8000:8000 \
+  -e HF_HUB_OFFLINE=1 -e HF_HUB_DISABLE_TELEMETRY=1 \
+  -v "$PWD/model-v2:/model:ro" \
+  vllm/vllm-openai@sha256:61fc8a896b0a4fbbbdc063bc4b0dbc25ce98e02b5050c24aeb7830ac02039b14 \
+  /model --served-model-name kdl-frontier-parser-nano \
+  --host 0.0.0.0 --port 8000 --max-model-len 8192 --max-num-seqs 24 \
+  --trust-remote-code --limit-mm-per-prompt '{"image": 1}'
+```
+
+## Run the complete official pipeline
+
+In a second terminal at that checkout, after `/v1/models` is ready:
+
+```bash
+env -i PATH="$PATH" HOME="$HOME" LANG=C.UTF-8 \
+  LLAMACLOUD_BENCH_LLM_NORMALIZATION=off HF_HUB_OFFLINE=1 \
+  KDL_NANO_ENDPOINT_URL=http://127.0.0.1:8000/v1 \
+  .venv/bin/parse-bench run kdl_frontier_nano \
+  --input_dir data --output_dir output-nano-v2-clean --open_report=False
+```
+
+No provider, prompt, normalization, scoring, layout adapter or aggregation edits are required. Do not use local provider-rule patches or a group-wise model ensemble. The server alias is intentionally the upstream default; the weights are the pinned v2 repository above.
+
+Read the five `_evaluation_report.json` reports, checking their coverage/failures as well as the process exit. Default metrics are Tables `grits_trm_composite`, Charts `rule_pass_rate`, Content `content_faithfulness`, Formatting `semantic_formatting`, Layout `layout_element_rule_pass_rate`; Overall is their unrounded five-way mean times 100. Use the official test-case loader's grouped, eligible case IDs rather than JSONL rule-row counts to verify coverage. The reports' `skipped` counters are additional scanned records outside `total_examples`, not additional successful eligible cases. The accompanying JSON keeps the observed single-pass values and counts, without redistributing benchmark inputs or predictions.