| 1 | # Reasonix Benchmarks |
| 2 | |
| 3 | Two primary harnesses live under `benchmarks/`; `cmd/e2ebench` also exposes a |
| 4 | SWE-bench Verified mode: |
| 5 | |
| 6 | - `e2e/` — the committed end-to-end task suite, driven by |
| 7 | [`cmd/e2ebench`](../cmd/e2ebench/main.go). It runs each task against a real |
| 8 | provider and emits a markdown + JSON report (accuracy, cache-hit rate, token |
| 9 | use, cost) suitable for pasting into a PR. |
| 10 | - `context-maintenance-e2e/` — a standalone seed → resume → comprehension |
| 11 | harness that A/B-compares cold-restart cache behavior with and without |
| 12 | context pruning. |
| 13 | |
| 14 | ## Directory layout |
| 15 | |
| 16 | ```text |
| 17 | benchmarks/ |
| 18 | ├── e2e/ |
| 19 | │ └── tasks/ |
| 20 | │ ├── compaction/ # task.toml + verify.sh + workdir/ seed |
| 21 | │ ├── fix-add-bug/ |
| 22 | │ ├── fizzbuzz/ |
| 23 | │ ├── palindrome/ |
| 24 | │ └── subagent-delegation/ |
| 25 | ├── swebench/ |
| 26 | │ ├── select_subset.py # helper for choosing evaluation instances |
| 27 | │ └── subset.json # committed SWE-bench Verified subset |
| 28 | └── context-maintenance-e2e/ |
| 29 | ├── main.go |
| 30 | └── run/ # state dir written by seed/resume (default) |
| 31 | ``` |
| 32 | |
| 33 | Each task under `e2e/tasks/<id>/` contains: |
| 34 | |
| 35 | | File | Purpose | |
| 36 | | --- | --- | |
| 37 | | `task.toml` | The task definition (prompt, step/timeout limits). | |
| 38 | | `verify.sh` | The grader: exits 0 iff the agent's artifacts are correct. | |
| 39 | | `workdir/` | Optional seed workspace, copied into the temp run dir before the agent starts. | |
| 40 | |
| 41 | ## task.toml schema |
| 42 | |
| 43 | `e2ebench` reads `benchmarks/e2e/tasks/<id>/task.toml` with the BurntSushi TOML |
| 44 | decoder. The task ID is the directory name; tasks run in sorted ID order. |
| 45 | |
| 46 | | Key | Type | Required | Description | |
| 47 | | --- | --- | --- | --- | |
| 48 | | `prompt` | string | yes | The task instruction handed to the agent. | |
| 49 | | `max_steps` | int | yes | Agent tool-call cap; passed through as `--max-steps` to `reasonix run`. | |
| 50 | | `timeout_sec` | int | no | Per-task wall-clock timeout in seconds; defaults to `240` when omitted or `0`. | |
| 51 | |
| 52 | Example (`tasks/fizzbuzz/task.toml`): |
| 53 | |
| 54 | ```toml |
| 55 | prompt = "Create a file named fizzbuzz.py containing a function fizzbuzz(n) that returns the string 'Fizz' when n is divisible by 3, 'Buzz' when divisible by 5, 'FizzBuzz' when divisible by both 3 and 5, and otherwise the number as a string. Do not print anything at import time." |
| 56 | max_steps = 12 |
| 57 | timeout_sec = 180 |
| 58 | ``` |
| 59 | |
| 60 | ## verify.sh contract |
| 61 | |
| 62 | `verify.sh` is the grader for a task: |
| 63 | |
| 64 | - It is a `bash` script run with `set -e`; exit code `0` means the task passed. |
| 65 | - It runs inside the temp work dir **after** the agent finishes, alongside the |
| 66 | copied `workdir/` seed and whatever files the agent produced — so it can |
| 67 | import generated Python modules, read `answer.txt`/`result.txt`, etc. |
| 68 | - The harness copies `verify.sh` into the work dir only after the run, so the |
| 69 | agent can never read the answer key during the run. |
| 70 | - Its stdout/stderr is streamed to the job log (stderr), not the report. |
| 71 | |
| 72 | Examples: `compaction/verify.sh` normalizes `answer.txt` (strip whitespace, |
| 73 | lowercase) and compares it to the expected `aldermoor-verrin`; |
| 74 | `fizzbuzz/verify.sh` imports the generated module and asserts on |
| 75 | `fizzbuzz(3)`, `fizzbuzz(5)`, `fizzbuzz(15)`, `fizzbuzz(7)`. |
| 76 | |
| 77 | ## Running the e2e suite |
| 78 | |
| 79 | Prerequisites: a `reasonix` binary (or `go run ./cmd/reasonix` …) with a |
| 80 | configured provider. The harness invokes the agent as |
| 81 | `reasonix run --auto --metrics <path> [--model NAME] [--max-steps N] [--profile delivery] [--ablate ARM] <prompt>` |
| 82 | inside a temp copy of the task's `workdir/`; the `--auto` flag is deliberate so |
| 83 | unattended fixture writes are allowed. |
| 84 | |
| 85 | ```sh |
| 86 | # Run the committed suite, report to stdout |
| 87 | go run ./cmd/e2ebench |
| 88 | |
| 89 | # Same suite with the delivery prompt profile |
| 90 | go run ./cmd/e2ebench -profile delivery |
| 91 | |
| 92 | # Write the markdown report to a file and the raw results to JSON |
| 93 | go run ./cmd/e2ebench -out report.md -json report.json |
| 94 | |
| 95 | # Grade a PR's diff (generates tests for the diff, grades with the repo's tests) |
| 96 | go run ./cmd/e2ebench -mode diff -base origin/main-v2 -repo . -attempts 3 -timeout 1800 |
| 97 | ``` |
| 98 | |
| 99 | The markdown report contains the solved count, cost/tokens per solved task, |
| 100 | median wall time, cache-hit rate, and a per-task table with failure class |
| 101 | (`solved`, `timeout`, `wrong_patch`, `no_metrics`, `skipped`, or the agent's |
| 102 | own outcome). |
| 103 | |
| 104 | ### Flags |
| 105 | |
| 106 | | Flag | Default | Purpose | |
| 107 | | --- | --- | --- | |
| 108 | | `-mode` | `suite` | `suite` \| `diff` \| `swebench` (`diff` generates tests for the PR diff; `swebench` runs the official per-instance evaluation) | |
| 109 | | `-suite` | `benchmarks/e2e` | Suite root (must contain `tasks/<id>/`). | |
| 110 | | `-bin` | `reasonix` | Path to the reasonix binary. | |
| 111 | | `-model` | *(config default)* | Provider/model name. | |
| 112 | | `-profile` | `baseline` | Prompt profile: `baseline` \| `delivery`. `delivery` appends `--profile delivery` to the agent invocation. | |
| 113 | | `-ablate` | *(none)* | Ablation arm: comma-separated subsystems to switch off — `evidence`, `planner`, `subagent`, `retrieval`, `compaction`; `none` \| `all`. | |
| 114 | | `-out` | *(stdout)* | Write the markdown report here. | |
| 115 | | `-json` | *(none)* | Write the JSON report here (optional). | |
| 116 | | `-budget` | `800000` | Abort once total tokens cross this (`0` = no cap). Remaining tasks are reported as skipped. | |
| 117 | |
| 118 | Diff-mode flags: |
| 119 | |
| 120 | | Flag | Default | Purpose | |
| 121 | | --- | --- | --- | |
| 122 | | `-repo` | `.` | Repo root (diff mode). | |
| 123 | | `-base` | *(none)* | Base ref to diff the PR head against (diff mode). | |
| 124 | | `-test-cmd` | `go test` | Grader command run on the affected packages (diff mode). | |
| 125 | | `-max-steps` | `80` | Agent tool-call cap for the diff task. | |
| 126 | | `-timeout` | `1200` | Agent timeout in seconds (diff mode). | |
| 127 | | `-attempts` | `1` | Diff mode: retry up to N times until a run passes (stochastic agent). | |
| 128 | |
| 129 | ## SWE-bench Verified mode |
| 130 | |
| 131 | `e2ebench` can also run the agent inside the official SWE-bench evaluation |
| 132 | images and hand the resulting patches to the official grader: |
| 133 | |
| 134 | ```sh |
| 135 | # Requires Docker, the `swebench` Python package, evaluation images, and a |
| 136 | # network/proxy setup that prevents the agent from reading upstream fixes. |
| 137 | go run ./cmd/e2ebench -mode swebench \ |
| 138 | -subset benchmarks/swebench/subset.json \ |
| 139 | -network reasonix-eval -proxy http://127.0.0.1:8080 |
| 140 | ``` |
| 141 | |
| 142 | SWE-bench mode accepts the `-model`, `-profile`, `-ablate`, `-permission`, |
| 143 | `-workers`, `-dataset`, `-run-id`, `-harness-python`, and `-keep-images` flags; |
| 144 | its report is produced by the official harness rather than the suite JSON |
| 145 | writer. |
| 146 | |
| 147 | ## Adding a new task |
| 148 | |
| 149 | 1. Create `benchmarks/e2e/tasks/<task-id>/`. |
| 150 | 2. Write `task.toml` with `prompt`, `max_steps`, and `timeout_sec` (see |
| 151 | [schema](#tasktoml-schema)). |
| 152 | 3. If the task needs seed files, add them under `workdir/` (they are copied |
| 153 | into the temp run dir; symlinks are skipped). |
| 154 | 4. Write `verify.sh`: `set -e`, exit 0 iff the agent's artifacts are correct. |
| 155 | Keep the expected answer out of the prompt and seed; the script runs in the |
| 156 | work dir and may validate anything the agent produced. |
| 157 | 5. Run the suite. Since `e2ebench` has no single-task filter, iterate against a |
| 158 | scratch suite root: |
| 159 | |
| 160 | ```sh |
| 161 | mkdir -p scratch/tasks/<task-id> |
| 162 | cp -r benchmarks/e2e/tasks/<task-id>/* scratch/tasks/<task-id>/ |
| 163 | go run ./cmd/e2ebench -suite scratch |
| 164 | ``` |
| 165 | |
| 166 | When it passes, remove the scratch dir and commit the task. |
| 167 | |
| 168 | ## context-maintenance-e2e |
| 169 | |
| 170 | This harness measures what happens when a long session goes idle past the |
| 171 | provider's cache TTL and then resumes: it A/B-compares cold-restart miss tokens |
| 172 | with and without pruning, and checks that the agent re-reads a file behind a |
| 173 | prune placeholder instead of hallucinating. |
| 174 | |
| 175 | It is hardcoded to the `deepseek-v4-flash` model at `https://api.deepseek.com` |
| 176 | and requires the `DEEPSEEK_API_KEY` environment variable. |
| 177 | |
| 178 | ```sh |
| 179 | export DEEPSEEK_API_KEY=... |
| 180 | |
| 181 | # Seed both arms (pruned + control) with a large session and warm the cache |
| 182 | go run ./benchmarks/context-maintenance-e2e seed |
| 183 | |
| 184 | # Wait past the provider's cache TTL, then resume: prune the "pruned" arm and |
| 185 | # compare cold-restart miss tokens |
| 186 | go run ./benchmarks/context-maintenance-e2e resume |
| 187 | |
| 188 | # Run the comprehension trials (agent must re-read a pruned file and answer |
| 189 | # from it); exits non-zero unless every trial passes |
| 190 | go run ./benchmarks/context-maintenance-e2e comprehension |
| 191 | ``` |
| 192 | |
| 193 | | Flag | Default | Purpose | |
| 194 | | --- | --- | --- | |
| 195 | | `-dir` | `benchmarks/context-maintenance-e2e/run` | State directory for `seed`/`resume` (sessions + `meta.json`, `resume-<ts>.json`). | |
| 196 | | `-trials` | `5` | Number of comprehension trials. | |
| 197 | |
| 198 | ## See also |
| 199 | |
| 200 | - [`docs/CLI.md`](../docs/CLI.md) — the `reasonix run` flags the e2e harness |
| 201 | passes through (`--auto`, `--metrics`, `--model`, `--max-steps`, |
| 202 | `--profile`, `--ablate`). |
| 203 | - [`cmd/e2ebench/main.go`](../cmd/e2ebench/main.go) — suite runner and report |
| 204 | renderer. |
| 205 |