| 1 | --- |
| 2 | name: best-of-n |
| 3 | description: Generate a small set of independent candidate solutions in worktrees, judge them against one explicit rubric, and apply the winner only after PASS verification. |
| 4 | metadata: |
| 5 | short-description: Compare independent candidates |
| 6 | --- |
| 7 | |
| 8 | # Best of N |
| 9 | |
| 10 | Use this skill when a consequential design, implementation, explanation, or |
| 11 | debugging task has several plausible solutions and comparison is worth the |
| 12 | extra model work. In Operate mode this is the preferred ensemble pattern for |
| 13 | high-stakes or ambiguous approaches. Do not use it for a tiny change or when |
| 14 | the user has already chosen the approach. |
| 15 | |
| 16 | ## Set The Tournament |
| 17 | |
| 18 | 1. Define one task, one evidence packet, and one explicit scoring rubric before |
| 19 | launching candidates. Include correctness, fit to the request, simplicity, |
| 20 | risk, and verification. |
| 21 | 2. Choose `N` from 2 to 4 for a quick comparison (default 3). For an explicit |
| 22 | experimental search, use the Workflow search option: 2–16 live candidates, |
| 23 | with larger validated populations queued at the Workflow host's 16-worker |
| 24 | concurrency gate rather than launched at once. |
| 25 | 3. Give every candidate the same task and rubric. Add only a candidate number; |
| 26 | do not steer candidates toward different conclusions unless diversity is an |
| 27 | explicit part of the request. |
| 28 | 4. Prefer a session goal (`create_goal` or active `/goal`) when the tournament |
| 29 | spans more than one parent turn. |
| 30 | |
| 31 | ## Generate Independently |
| 32 | |
| 33 | Start the candidates as parallel background `agent` workers and return agent_ids |
| 34 | immediately so the parent stays free. For proposals, reviews, or research, keep |
| 35 | them read-only: |
| 36 | |
| 37 | ```json |
| 38 | { |
| 39 | "action": "start", |
| 40 | "name": "candidate_1", |
| 41 | "prompt": "Produce candidate 1 for the task below. Return the proposal, evidence, risks, and rubric self-score. Do not edit files.\n\n<TASK AND RUBRIC>", |
| 42 | "type": "worker", |
| 43 | "model_strength": "same", |
| 44 | "write_authority": "read_only" |
| 45 | } |
| 46 | ``` |
| 47 | |
| 48 | Launch the remaining candidates with the same contract, then use `agent` wait |
| 49 | or completion events to collect every result. Do not show one candidate another |
| 50 | candidate's answer before generation finishes. |
| 51 | |
| 52 | When candidates must implement code, give each one: |
| 53 | |
| 54 | - `type: "builder"` |
| 55 | - `worktree: true` |
| 56 | - `write_authority: "worktree_write"` |
| 57 | - the same bounded `write_roots` or `exact_files` |
| 58 | |
| 59 | Never run parallel writers in the parent checkout. Each builder must return |
| 60 | the structured candidate contract (candidate id, hypothesis, paths, commands, |
| 61 | self-verdict, risks, and artifact references). A self-verdict is evidence to |
| 62 | inspect, not a hard-gate result. |
| 63 | |
| 64 | Optional diversity: pin different `model` / Fleet `fleet_profile` values when |
| 65 | the project has multiple capable routes; otherwise keep model strength `same`. |
| 66 | |
| 67 | ## Judge Once |
| 68 | |
| 69 | Use one read-only reviewer worker, or the parent when the result is small, to |
| 70 | score all candidates against the original rubric. The judge must: |
| 71 | |
| 72 | - cite evidence from each candidate rather than vote by style; |
| 73 | - reject candidates that violate authority, scope, or verification gates; |
| 74 | - treat candidate-reported commands and PASS claims as untrusted until replay; |
| 75 | - name the winner and the decisive reasons; |
| 76 | - identify useful pieces worth combining, if any; |
| 77 | - say when the candidates are tied or all fail. |
| 78 | |
| 79 | Do not ask candidates to vote for themselves. Do not silently merge incompatible |
| 80 | approaches into a new unreviewed solution. |
| 81 | |
| 82 | ## Integrate Only After PASS |
| 83 | |
| 84 | For proposal-only work, return the winning answer with a compact score summary. |
| 85 | For code work: |
| 86 | |
| 87 | 1. Freeze the baseline, evaluator, hard gates, score rule, and authority before |
| 88 | a larger search admits candidates. Any evaluator change starts a revision. |
| 89 | 2. After a worker loses write authority, apply its patch to a clean baseline |
| 90 | and let the runtime—not that worker—run hard gates and scoring. |
| 91 | 3. Inspect the winning worktree diff and independently replay it on the clean |
| 92 | baseline. A different read-only model may look for gaming, but deterministic |
| 93 | tests remain the authority. |
| 94 | 4. Present the verified winner for review. Applying or merging is a separate, |
| 95 | explicit user action; `NONE` is valid when every candidate fails. |
| 96 | 5. Preserve losing and failed candidate receipts as useful negative results. |
| 97 | |
| 98 | The checked-in `operate_best_of_n.workflow.js` recipe supports |
| 99 | `strategy: "search"` for structured 2–16 candidate generation and review. It |
| 100 | does **not** yet turn prompt-listed commands into hidden runtime gates. Do not |
| 101 | advertise those gates until the runtime evaluator host consumes a frozen |
| 102 | `WorkflowSearchSpec`. |
| 103 | |
| 104 | Stop early when one candidate reveals a hard constraint that invalidates the |
| 105 | tournament. Report the negative result rather than spending the remaining |
| 106 | budget to manufacture variety. |
| 107 |