ybelkada HF Staff commited on
Commit
361aaec
·
verified ·
1 Parent(s): b176a2d

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -9
README.md CHANGED
@@ -48,7 +48,7 @@ changed outcome. Tool calls are compared over every task, and on the tasks both
48
  - **SWE-bench Pro (hard 51):** the 51 hardest tasks of [SWE-bench Pro](https://huggingface.co/datasets/harness-hub/swebench-pro-v2-hard51-harbor), 2 h and 100 agent steps per task.
49
 
50
  Step 160 was picked by looking at the checkpoints' scores on SWE-bench Pro and on an 18-task subset of Terminal-Bench
51
- 2.1, tracked during training. Those two results are optimistic; the Terminal-Bench 2.1 row without the subset is not.
52
 
53
  ## Use with pi
54
 
@@ -145,12 +145,4 @@ The adapter must be loaded on top of Qwen3.8-27B. It does not work with other ba
145
  The training script is
146
  [`examples/async_grpo_harbor`](https://github.com/huggingface/trl/tree/main/examples/async_grpo_harbor) in TRL.
147
 
148
- ## Limitations
149
 
150
- - **Small benchmarks.** With 51 to 89 tasks and one attempt per task, the 95% interval on a paired difference is
151
- ±10 points or more. The sign test counts only the tasks that changed outcome.
152
- - **Thinking off.** The adapter was trained and evaluated without thinking. With thinking on, expect different behaviour.
153
- - **Timeouts count as failures.** A rollout past the time or step limit is graded 0, for the base and the adapter alike.
154
- - **One harness.** The adapter was trained inside pi's prompts and tools. With another agent, expect it to
155
- behave like the base model, or worse; [harness-hub/Qwen3.8-27B-multi-harness](https://huggingface.co/harness-hub/Qwen3.8-27B-multi-harness) is one adapter
156
- for pi, Hermes and OpenCode.
 
48
  - **SWE-bench Pro (hard 51):** the 51 hardest tasks of [SWE-bench Pro](https://huggingface.co/datasets/harness-hub/swebench-pro-v2-hard51-harbor), 2 h and 100 agent steps per task.
49
 
50
  Step 160 was picked by looking at the checkpoints' scores on SWE-bench Pro and on an 18-task subset of Terminal-Bench
51
+ 2.1, tracked during training.
52
 
53
  ## Use with pi
54
 
 
145
  The training script is
146
  [`examples/async_grpo_harbor`](https://github.com/huggingface/trl/tree/main/examples/async_grpo_harbor) in TRL.
147
 
 
148