| 1 | # Agent Fleet |
| 2 | |
| 3 | Agent Fleet is the local-first control plane for durable multi-worker runs. It |
| 4 | is **not** a separate execution engine: a fleet worker is a headless |
| 5 | `codewhale exec` run that the fleet launches and tracks durably. See |
| 6 | [AGENT_RUNTIME.md](AGENT_RUNTIME.md) for how sub-agents, `exec`, and the fleet |
| 7 | converge on one durable runtime. In product language, a user may still "open a |
| 8 | sub-agent"; in architecture language, durable nested work should be a |
| 9 | fleet-backed worker with a role. |
| 10 | |
| 11 | Use Fleet rather than short-lived `agent` fanout whenever the work |
| 12 | needs retry, sleep/restart survival, remote execution, receipts, or a ledgered |
| 13 | audit trail. The initial CLI surface is: |
| 14 | |
| 15 | For a guided start-to-monitor walkthrough that combines Fleet task specs with |
| 16 | Workflow authoring, see [Fleet + Workflow Tutorial](FLEET_WORKFLOW_TUTORIAL.md). |
| 17 | |
| 18 | ```sh |
| 19 | codewhale fleet init |
| 20 | codewhale fleet run tasks.json --max-workers 4 |
| 21 | codewhale fleet status |
| 22 | codewhale fleet inspect <worker-id> |
| 23 | codewhale fleet logs <worker-id> |
| 24 | codewhale fleet artifacts <worker-id> |
| 25 | codewhale fleet interrupt <worker-id> |
| 26 | codewhale fleet restart <worker-id> |
| 27 | codewhale fleet resume <run-id> |
| 28 | codewhale fleet stop --all |
| 29 | ``` |
| 30 | |
| 31 | `codewhale fleet resume <run-id>` is the restart-recovery verb: it replays the |
| 32 | ledger, reconciles any in-flight lease whose worker stopped heartbeating |
| 33 | (retrying within the task's budget, else failing and escalating per the alert |
| 34 | policy), and prints the post-resume status. It launches no new work and is |
| 35 | idempotent, so it is safe to run after a manager exit, laptop sleep, or runtime |
| 36 | restart. |
| 37 | |
| 38 | Fleet state is stored under the workspace in `.codewhale/fleet.jsonl`. Worker |
| 39 | logs and adapter logs are stored under `.codewhale/fleet/` and |
| 40 | `.codewhale/fleet-host/`. |
| 41 | |
| 42 | ### Interactive and persistent status |
| 43 | |
| 44 | `/fleet status` and `codewhale fleet status` are the **same** command on two |
| 45 | surfaces. Both read the durable `.codewhale/fleet.jsonl` ledger for the |
| 46 | workspace, through one shared control-plane contract, and both report the same |
| 47 | verb id (`fleet.status`), read-vs-write authority, persistence scope, and |
| 48 | receipt. When the workspace has no ledger they say so with a typed reason |
| 49 | (`no_fleet_ledger`) instead of rendering an empty-looking "all clear" — and |
| 50 | neither creates the ledger as a side effect of reading it. |
| 51 | |
| 52 | The current interactive session's sub-agents are a **different set**, and now |
| 53 | have their own name: |
| 54 | |
| 55 | - `/fleet workers` (or `/subagents`, or `n`) shows sub-agents attached to the |
| 56 | current TUI session. It does not read the persistent ledger. |
| 57 | - `/fleet list|status|interrupt|resume` and `codewhale fleet |
| 58 | list|status|interrupt|resume` act on the durable ledger. |
| 59 | - `codewhale fleet restart <worker-id>` is CLI-only: it re-leases the task and |
| 60 | then drives the manager loop to completion. `/fleet restart` does not |
| 61 | silently do a smaller thing — it reports `surface_not_supported` and names |
| 62 | the CLI command. |
| 63 | |
| 64 | Before v0.9.2, `/fleet status` showed session sub-agents. That reading is gone; |
| 65 | `/fleet workers` replaces it. |
| 66 | |
| 67 | The contract behind this — descriptors, availability reasons, exact-identity |
| 68 | targets, receipts, typed unknowns, and bounds — is documented in |
| 69 | [`docs/COMMAND_CONTROL_PLANE.md`](COMMAND_CONTROL_PLANE.md). |
| 70 | |
| 71 | ## Authoring agent profiles (`/fleet setup`) |
| 72 | |
| 73 | `/fleet setup` (also `/fleet setup edit` / `new`) opens an in-TUI wizard for |
| 74 | authoring a reusable agent-team profile. Bare `/fleet` and the |
| 75 | `roster`/`roles`/`profiles`/`party` aliases open the roster (the saved profiles). |
| 76 | `/fleet workers` opens the current-session worker view; `/subagents` is a |
| 77 | compatibility shortcut for that view. For durable run history, use |
| 78 | `/fleet status` or the shell command `codewhale fleet status` described above — |
| 79 | they are the same command. |
| 80 | |
| 81 | The wizard is progressive: you make one focused choice at a time — a **role**, |
| 82 | then a **model** (`inherit`, or a concrete model from *any configured |
| 83 | provider*, not only the one the parent session is currently using), then a |
| 84 | **thinking tier** (`inherit`, `off`, `low`, `medium`, `high`, `max`, or `auto`) |
| 85 | — and then review the full posture (route, thinking, permissions, tools, |
| 86 | scope, and review policy) before doing anything. On Review, press **`s`** to |
| 87 | choose where the profile lives: |
| 88 | |
| 89 | - **Project** (default) writes `.codewhale/agents/<role>.toml` and can travel |
| 90 | with this repository. |
| 91 | - **Personal** writes `$CODEWHALE_HOME/agents/<role>.toml` and is available in |
| 92 | every repository on this machine. A project profile with the same id still |
| 93 | overrides the personal profile for that project. |
| 94 | |
| 95 | Profile scope controls where a role definition is reusable; it does not widen |
| 96 | the authority of a running operation. To coordinate several nearby |
| 97 | repositories, start Codewhale from their shared parent directory so that parent |
| 98 | is the workspace. Explicit trusted external paths or Full Access can still |
| 99 | change what tools may reach; workers inherit the active trust and permission |
| 100 | posture, never the profile's storage scope. |
| 101 | |
| 102 | Picking a concrete model pins its provider explicitly: the saved profile records both |
| 103 | `model` and `provider` fields, so the route it names doesn't depend on |
| 104 | whichever provider happens to be active when the profile is later loaded. |
| 105 | Pressing **Enter** ("start") on the review step previews the exact starter |
| 106 | profile TOML inline on that same screen; nothing is written until you save it. |
| 107 | The `provider` field may be a built-in provider id such as `openrouter` or a |
| 108 | user-named OpenAI-compatible provider configured under `[providers.<name>]` |
| 109 | such as `lm-studio`; the launch path preserves that id and fails closed if the |
| 110 | provider is not configured. |
| 111 | |
| 112 | When a provider is configured, the review step also offers model-assisted |
| 113 | drafting behind an explicit preview-before-save gate: |
| 114 | |
| 115 | - Press **`m`** to have your first configured model draft the profile. The |
| 116 | draft arrives sanitized and bounded — permissions stay at the **fleet floor** |
| 117 | (no shell, no trust, approval required) regardless of what the model |
| 118 | proposes. |
| 119 | - **Drafting is not saving.** The exact rendered TOML preview renders |
| 120 | inline on the review step (not in a separate scrollable viewer), so nothing |
| 121 | is saved until you press **`g`** or **Enter** to save (or press `m` again |
| 122 | to redraft). Saving writes the profile to the project or personal scope |
| 123 | shown in the preview. |
| 124 | |
| 125 | ## Naming: Modes, Workflow, and Fleet |
| 126 | |
| 127 | These names describe different layers, not competing systems. Plan and Act are |
| 128 | the everyday work modes. Operate accepts ordinary messages and keeps the |
| 129 | parent's normal tool surface under the same approval, sandbox, shell, ask-rule, |
| 130 | and repository protections as Act. It prefers background Fleet workers for |
| 131 | independent, parallel, isolated, or long-running work, but does not require a |
| 132 | worker for every executable step. Workflow is an optional orchestration overlay |
| 133 | for work that needs ordering, gates, shared budgets, replay, or deterministic |
| 134 | fan-in. |
| 135 | |
| 136 | The short public vocabulary is: |
| 137 | |
| 138 | - **Fleet** = who does the work: the configured workers, roles, models, hosts, |
| 139 | and trust boundaries. |
| 140 | - **Workflow** = what order the work follows: phases, gates, budgets, replay, |
| 141 | and fan-in. |
| 142 | - **Lane** = one running Workflow instance and its live progress. |
| 143 | - **Runtime** = where and how a Lane executes: local or remote process, |
| 144 | provider route, sandbox, and API boundary. |
| 145 | |
| 146 | - **Workflow** is the repeatable plan and user-facing orchestration |
| 147 | overlay: a script/IR that decides which phases and agents run next, keeps |
| 148 | intermediate results out of the main conversation, and can be inspected or |
| 149 | rerun. A Workflow run should have a visible progress view and a clear active |
| 150 | header state instead of feeling like a hidden background task. |
| 151 | - **Fleet** is the durable sub-agent configuration and execution substrate: |
| 152 | slots, profiles, per-slot models, tool posture, local/SSH hosts, trust |
| 153 | policy, leases, heartbeats, logs, receipts, and status APIs. |
| 154 | - **High fan-out** is a behavior of a Workflow run, not a separate system: |
| 155 | when a phase needs many workers at once, Workflow dispatches them as a |
| 156 | Fleet-backed run (durable workers, receipts, goal re-dispatch) rather than |
| 157 | reviving prompt-only sub-agent fanout. |
| 158 | - **Fan-in is explicit:** when the user needs one combined result, an owner |
| 159 | aggregates, verifies, and synthesizes the worker receipts. Independent tasks |
| 160 | may finish separately; dispatch is never presented as completion. |
| 161 | |
| 162 | UI guidance: keep the main transcript calm. A Workflow run should appear as a |
| 163 | compact progress card plus work-bar rows (the strip above the transcript, or |
| 164 | a side rail) with phase names, worker counts, receipts, and nested |
| 165 | indentation for child workers. Use the whale mark sparingly as an active |
| 166 | header/status signal; avoid repeating emoji-heavy rows for every worker. |
| 167 | |
| 168 | ## Exact Fleets and the Reasoning Router |
| 169 | |
| 170 | An exact Fleet freezes the provider, model, reasoning policy, and permission |
| 171 | ceiling of every worker before a Workflow starts. Save it as |
| 172 | `fleets/<name>.toml` in the workspace or under `$CODEWHALE_HOME`. Models cannot |
| 173 | replace those assignments at runtime: |
| 174 | |
| 175 | ```toml |
| 176 | name = "release" |
| 177 | schema = "exact" |
| 178 | schema_revision = 1 |
| 179 | reasoning_router = "luna-low" |
| 180 | |
| 181 | [[members]] |
| 182 | id = "implementer" |
| 183 | role = "builder" |
| 184 | provider = "zai" |
| 185 | model = "glm-5.2" |
| 186 | reasoning = "auto" |
| 187 | permissions = "read_write" |
| 188 | |
| 189 | [[members]] |
| 190 | id = "advice" |
| 191 | role = "consultant" |
| 192 | provider = "openai" |
| 193 | model = "gpt-5.6" |
| 194 | reasoning = "high" |
| 195 | permissions = "read_only" |
| 196 | ``` |
| 197 | |
| 198 | The optional Reasoning Router is a reusable service, not a Fleet member. Save |
| 199 | one profile at `routers/<name>.toml` in either search root and reference it from |
| 200 | any number of Fleets: |
| 201 | |
| 202 | ```toml |
| 203 | name = "luna-low" |
| 204 | schema = "reasoning_router" |
| 205 | schema_revision = 1 |
| 206 | provider = "openai" |
| 207 | model = "gpt-5.6-luna" |
| 208 | call_reasoning = "low" |
| 209 | ``` |
| 210 | |
| 211 | At runtime it may choose only the reasoning tier for an already-frozen worker |
| 212 | route. It cannot change the worker, provider, model, role, tools, or permissions. |
| 213 | The Router call itself is capped at `off` or `low`; more expensive values are |
| 214 | rejected. A manually selected worker reasoning tier makes no Router call. Route |
| 215 | and reasoning receipts name the worker model and, when used, the Router's exact |
| 216 | provider/model so the operator can see which model did which job. If the same |
| 217 | bare Router or Fleet name exists in both roots, qualify it as |
| 218 | `workspace/<name>` or `codewhale_home/<name>` instead of relying on shadowing. |
| 219 | |
| 220 | Each member's `permissions` preset is a **ceiling**, never a grant: it is |
| 221 | intersected with the live session posture, so a `read_write` member inside a |
| 222 | read-only session runs read-only. The intersection becomes the child's real tool |
| 223 | envelope — `permissions = "none"` leaves it with no tools at all, and a member |
| 224 | without a network tool loses every web, fetch, browse, `web.run`, and MCP |
| 225 | surface rather than merely being refused at call time. A member that cannot |
| 226 | write loses the mutating file tools, and — when it kept `shell = "full"` so it |
| 227 | can run checks — the raw shell as well, keeping the bounded verification tools |
| 228 | (`run_tests`, `run_verifiers`) it exists for: an arbitrary shell command mutates |
| 229 | a workspace just as surely as `write_file`, so leaving it would make |
| 230 | `write = false` untrue. That verification surface is bounded only in its |
| 231 | *default* form, so the unbounded arguments go with the shell: a write-denied |
| 232 | member may run the built-in gates, but not `run_verifiers` with an explicit |
| 233 | `commands` array or `run_tests` with a raw `args` string, both of which spawn |
| 234 | operator-supplied programs and are the raw shell by another name. |
| 235 | |
| 236 | Two of these denials are narrower than a tool name, because two tools reach past |
| 237 | their name. `rlm` loads a `url` by calling the fetch tool *inside the process*, |
| 238 | under its own name, and `rlm`'s `eval` action runs Python against a live kernel |
| 239 | — sockets and filesystem both. So the family is denied **per action**, not |
| 240 | wholesale: |
| 241 | |
| 242 | - **No network tool** removes `rlm_open` and `rlm_eval`, by the legacy alias and |
| 243 | by the `rlm {action: ...}` spelling alike. This is deliberately narrower than |
| 244 | the capability it protects: `rlm_open` picks its source from *input fields* |
| 245 | (`file_path`, `content`, `url`, `session_object`), not from the action name, |
| 246 | and the action-policy seam resolves names rather than field shapes — it cannot |
| 247 | prove a source is local before the tool runs. Rather than leave a URL-shaped |
| 248 | hole, a network-denied member loses RLM loading entirely, **including the |
| 249 | purely local `file_path` form**, and keeps only the bounded metadata actions |
| 250 | (`session_objects`, `configure`, `close`). |
| 251 | - **No write** removes `rlm_eval` only. Loading a large local file into a kernel |
| 252 | and reading it is analysis, not mutation, so the rest of the family stays. |
| 253 | |
| 254 | Beyond those names, a network-denied member is refused at call time whenever any |
| 255 | tool is handed a URL-bearing field (`url`, `urls`, `endpoint`, `target`, …) |
| 256 | holding an `http`/`https`/`ws`/`wss`/`ftp` address. A URL appearing in file |
| 257 | *content* or a search pattern is data, not a destination, and is untouched. |
| 258 | |
| 259 | A member's `role` picks the worker |
| 260 | posture (and system prompt) when that role already fits inside the ceiling — |
| 261 | `reviewer`, `verifier`, `consultant`, `planner` — and a domain-specific role |
| 262 | such as `auditor` falls back to the narrowest posture the ceiling allows. Tasks |
| 263 | cannot override any of it: `model`, `thinking`, |
| 264 | `subagent_type`, `allowed_tools`, and `write_authority` are rejected on an exact |
| 265 | Fleet rather than silently ignored. |
| 266 | |
| 267 | Reasoning receipts record the requested tier *and* the tier the provider was |
| 268 | actually asked for. Those differ whenever a route cannot express the requested |
| 269 | one — CodeWhale's route normalizer sends `high` for a requested `low` on most |
| 270 | routes, and Z.AI's GLM routes express only thinking on/off — so the receipt |
| 271 | reports the real request rather than the label that was selected. The value a |
| 272 | call actually carries is spelled by that route's own normalizer, not by the tier |
| 273 | label: an OpenAI Codex route is asked for `xhigh`, not `max`, and cannot be |
| 274 | asked for `off` at all. |
| 275 | |
| 276 | A receipt also keeps a member's **semantic role** and its **permission posture** |
| 277 | apart. `member_role` is what the operator named and what gates key on; |
| 278 | `posture_role` (present only when it differs) is the built-in tool surface the |
| 279 | clamped ceiling permits — so a member named `auditor` running under the `scout` |
| 280 | posture is displayed as `auditor` and enforced as `scout`, and neither fact is |
| 281 | substituted for the other. |
| 282 | |
| 283 | A Workflow start fails closed on anything decidable locally: an unresolvable |
| 284 | provider or model, a missing credential, a client that cannot be built for a |
| 285 | member's route, or an `auto` member with no usable Reasoning Router. Per-task |
| 286 | validation that the spawn boundary would refuse anyway — notably a write-capable |
| 287 | member with no declared `write_roots`/`exact_files`/`coordination_contracts` — |
| 288 | is checked before the Router is called, so an invalid task never spends a |
| 289 | routing request. If a spawn fails *after* a Router decision, the receipt is |
| 290 | still recorded: the tokens were spent, and any cross-provider disclosure already |
| 291 | happened. |
| 292 | |
| 293 | ## Manager-owned Workflow fan-in |
| 294 | |
| 295 | When parallel work must return one combined answer, use a manager-owned |
| 296 | Workflow instead of a flat `agent` fan-out: |
| 297 | |
| 298 | 1. **Cast one manager** (operator or workflow orchestrator). |
| 299 | 2. **Fan out** child tasks through `workflow` (`task()`, `parallel()`, |
| 300 | `pipeline()`, `phase()`) or a single manager session that owns the children. |
| 301 | 3. **Wait** for child receipts or completion events. |
| 302 | 4. **Aggregate and verify** load-bearing claims before treating them as facts. |
| 303 | 5. **Synthesize** one result the operator can depend on. |
| 304 | |
| 305 | Raw `agent` fan-out is appropriate only for independent, fire-and-forget work |
| 306 | where no single fan-in result is required. If results must be merged, compared, |
| 307 | or verified, route through `workflow` so the manager owns fan-in. |
| 308 | |
| 309 | ## Workflow on Fleet |
| 310 | |
| 311 | The intended high-capability path is agent-authored. When the main agent |
| 312 | decides a task needs more durable coordination than turn-by-turn sub-agent |
| 313 | calls, it drafts a Workflow script/IR, presents the run plan according to the |
| 314 | active permission mode, and the runtime compiles it into typed Fleet work. |
| 315 | |
| 316 | Fleet remains the sub-agent config surface. It owns slot count, role profiles, |
| 317 | saved route pins or inheritance, tool posture, launch concurrency, and the ledger. |
| 318 | Workflow owns only the orchestration plan: branch, sequence, loop, expand, |
| 319 | review, and reduce decisions. The workflow script must not get direct shell, |
| 320 | filesystem, network, provider-secret, cancellation, or TUI authority; workers |
| 321 | perform real work as `codewhale exec` processes. |
| 322 | |
| 323 | Default Workflow-to-Fleet validation is intentionally bounded: |
| 324 | |
| 325 | - 1,000 total worker agents per Workflow run; |
| 326 | - 16 live worker agents at once; larger populations queue (block) on the host's |
| 327 | per-run concurrency gate until a live slot frees, then route through Fleet; |
| 328 | - 5 recursive Fleet rings as the hard ceiling (default user configuration: 2); |
| 329 | - bounded loops only (`max_iterations` required); |
| 330 | - bounded dynamic expansion only (`max_children` plus a template required). |
| 331 | |
| 332 | These are population limits, not a demand to launch everything at once. A |
| 333 | 1,000-agent Workflow should still drain through the configured Fleet worker |
| 334 | pool. |
| 335 | Recommended model layouts, such as a DeepSeek Pro orchestrator with Flash |
| 336 | workers in the first ring and cheaper workers farther out, are presets only. |
| 337 | Every slot can inherit the active model or carry an explicit model override. |
| 338 | Inheritance is literal: the model you select in `/model` is the **operator** |
| 339 | (the pinned first row in `/fleet roster`), and any worker whose task spec and |
| 340 | roster profile pin no model runs on that session model. Task-level `model` and |
| 341 | profile `model` overrides still win; route receipts record which source |
| 342 | applied (`task.model`, `agent_profile.model`, or `run.model`). |
| 343 | |
| 344 | The setup UI should render this as an expanding grid: an orchestrator plus a |
| 345 | small number of visible sub-agent slots, with Right/Enter drilling into a slot's |
| 346 | next recursive ring rather than trying to show the whole tree at once. |
| 347 | |
| 348 | ## Task Spec |
| 349 | |
| 350 | `codewhale fleet run` accepts JSON or TOML. A minimal JSON spec: |
| 351 | |
| 352 | ```json |
| 353 | { |
| 354 | "name": "local smoke", |
| 355 | "tasks": [ |
| 356 | { |
| 357 | "id": "lint", |
| 358 | "name": "Lint", |
| 359 | "instructions": "Run the lint check and report failures.", |
| 360 | "expected_artifacts": ["log"] |
| 361 | } |
| 362 | ] |
| 363 | } |
| 364 | ``` |
| 365 | |
| 366 | Workers are optional. If omitted, Codewhale creates local worker slots up to |
| 367 | `--max-workers`. |
| 368 | |
| 369 | Task specs are typed in Rust and keep verification data separate from worker |
| 370 | transcripts. A task can declare: |
| 371 | |
| 372 | - `id`, `name`, `description`, `objective`, and `instructions` |
| 373 | - `worker` role, tool profile, tools, and required capabilities |
| 374 | - `workspace` root, required files, writable paths, and environment allowlist |
| 375 | - `input_files`, extra `context`, `budget`, `timeout_seconds`, and `retry_policy` |
| 376 | - `expected_artifacts`, `scorer`, `tags`, and free-form `metadata` |
| 377 | |
| 378 | Workers write bounded artifact files under `.codewhale/fleet/` and ledger only |
| 379 | the artifact refs: kind, path, checksum, MIME type, and size. Receipts record |
| 380 | `pass`, `fail`, `partial`, `skip`, or `timeout`; failed receipts may also mark |
| 381 | the source as `transport`, `task`, or `verifier`. `codewhale fleet status` |
| 382 | surfaces those failure-source counts separately. |
| 383 | |
| 384 | Deterministic built-in scorers are `exit_code`, `file_exists`, `regex_match`, |
| 385 | and `json_path`. Specs may also declare `command`, |
| 386 | `code_whale_verifier_prompt`, or `manual`; those record a partial receipt until |
| 387 | an explicit verifier pass completes. |
| 388 | |
| 389 | ### Using Role Presets |
| 390 | |
| 391 | Tasks can reference a role name, and the fleet manager fills in defaults |
| 392 | from the role registry. Built-in roles (`smoke-runner`, `reviewer`, `builder`, |
| 393 | `read-only`) are always available; define your own in `[fleet.roles]`. |
| 394 | |
| 395 | ```json |
| 396 | { |
| 397 | "name": "smoke check", |
| 398 | "tasks": [ |
| 399 | { |
| 400 | "id": "lint", |
| 401 | "name": "Lint check", |
| 402 | "instructions": "Run lint and report failures.", |
| 403 | "worker": { "role": "smoke-runner" }, |
| 404 | "expected_artifacts": ["log"] |
| 405 | } |
| 406 | ] |
| 407 | } |
| 408 | ``` |
| 409 | |
| 410 | The task inherits the role's tool profile, budget, and timeout. You can |
| 411 | override any field in the task spec: |
| 412 | |
| 413 | ```json |
| 414 | { |
| 415 | "id": "deep-review", |
| 416 | "name": "Deep review", |
| 417 | "instructions": "Review the entire crate for soundness issues.", |
| 418 | "worker": { |
| 419 | "role": "reviewer", |
| 420 | "tools": ["cargo", "rg", "git"], |
| 421 | "capabilities": ["rust"] |
| 422 | }, |
| 423 | "input_files": ["crates/**/*.rs"], |
| 424 | "budget": { "max_tokens": 32000 }, |
| 425 | "expected_artifacts": ["log", "report"], |
| 426 | "scorer": { "kind": "regex_match", "path": ".codewhale/fleet/report.md", "pattern": "finding|all clear" } |
| 427 | } |
| 428 | ``` |
| 429 | |
| 430 | ### Multi-Task Run Example |
| 431 | |
| 432 | A single fleet run can dispatch several independent tasks in parallel: |
| 433 | |
| 434 | ```json |
| 435 | { |
| 436 | "name": "CI gate", |
| 437 | "tasks": [ |
| 438 | { |
| 439 | "id": "check", |
| 440 | "name": "Compile check", |
| 441 | "instructions": "Run cargo check --workspace and report errors.", |
| 442 | "worker": { "role": "builder" }, |
| 443 | "expected_artifacts": ["log"], |
| 444 | "scorer": { "kind": "exit_code" } |
| 445 | }, |
| 446 | { |
| 447 | "id": "clippy", |
| 448 | "name": "Clippy lint", |
| 449 | "instructions": "Run cargo clippy --workspace and report warnings.", |
| 450 | "worker": { "role": "reviewer", "tools": ["cargo", "cargo-clippy"] }, |
| 451 | "expected_artifacts": ["log"], |
| 452 | "scorer": { "kind": "exit_code" } |
| 453 | }, |
| 454 | { |
| 455 | "id": "security", |
| 456 | "name": "Secret audit", |
| 457 | "instructions": "Search for plaintext secrets and report any matches.", |
| 458 | "worker": { "role": "read-only", "tools": ["rg"] }, |
| 459 | "input_files": ["crates/**/*.rs"], |
| 460 | "expected_artifacts": ["log", "report"], |
| 461 | "retry_policy": { "max_attempts": 1 } |
| 462 | } |
| 463 | ] |
| 464 | } |
| 465 | ``` |
| 466 | |
| 467 | ## Alerts |
| 468 | |
| 469 | Fleet alerting is disabled by default. A caller must supply an enabled alert |
| 470 | config before anything is sent. Routes match typed fleet event classes, not log |
| 471 | strings: |
| 472 | |
| 473 | - `stale` |
| 474 | - `restart_exhausted` |
| 475 | - `needs_human` |
| 476 | - `budget_exceeded` |
| 477 | - `verifier_failed` |
| 478 | - `run_completed` |
| 479 | |
| 480 | Adapter config stores environment variable names, not secret values. Send-time |
| 481 | code resolves those names from the environment or a future secrets provider. |
| 482 | Ledger records store only audit labels such as `slack`, `webhook`, or |
| 483 | `pagerduty`; task specs persisted in the ledger redact webhook URLs and routing |
| 484 | keys. |
| 485 | |
| 486 | Example alert config shape: |
| 487 | |
| 488 | ```json |
| 489 | { |
| 490 | "enabled": true, |
| 491 | "dry_run": true, |
| 492 | "routes": [ |
| 493 | { |
| 494 | "events": ["stale", "restart_exhausted", "verifier_failed"], |
| 495 | "adapter": "ops-slack" |
| 496 | }, |
| 497 | { |
| 498 | "events": ["restart_exhausted"], |
| 499 | "adapter": "pager" |
| 500 | } |
| 501 | ], |
| 502 | "adapters": { |
| 503 | "ops-slack": { |
| 504 | "kind": "slack", |
| 505 | "webhook_env": "CODEWHALE_FLEET_SLACK_WEBHOOK", |
| 506 | "channel": "#codewhale-fleet" |
| 507 | }, |
| 508 | "pager": { |
| 509 | "kind": "pager_duty", |
| 510 | "routing_key_env": "CODEWHALE_FLEET_PAGERDUTY_ROUTING_KEY", |
| 511 | "severity": "critical" |
| 512 | } |
| 513 | } |
| 514 | } |
| 515 | ``` |
| 516 | |
| 517 | Use dry-run to inspect a redacted adapter payload without sending: |
| 518 | |
| 519 | ```sh |
| 520 | codewhale fleet alert-dry-run \ |
| 521 | --event stale \ |
| 522 | --run-id fleet-demo \ |
| 523 | --worker-id fleet-demo-local-1 \ |
| 524 | --task-id release-triage \ |
| 525 | --reason "worker heartbeat stale since 2026-06-13T02:00:00Z" \ |
| 526 | --adapter slack |
| 527 | ``` |
| 528 | |
| 529 | The payload includes the run id, worker id, task id, status, short reason, and |
| 530 | safe inspection commands such as `codewhale fleet status` and |
| 531 | `codewhale fleet inspect <worker-id>`. Endpoints, webhook secrets, and |
| 532 | PagerDuty routing keys are shown as `<redacted:env:...>`. |
| 533 | |
| 534 | ## Status Surfaces |
| 535 | |
| 536 | `codewhale fleet status` shows compact counts for queued, running, completed, |
| 537 | partial, failed, restarted, escalated, cancelled, stale, and verifier/transport |
| 538 | failure sources. `inspect` shows the worker state plus the current task |
| 539 | objective, role, host, heartbeat, latest event, artifact refs, latest error, and |
| 540 | alert state. `logs` prints bounded log artifact contents, and `artifacts` lists |
| 541 | artifact refs without embedding large payloads. |
| 542 | |
| 543 | The Runtime API exposes the same ledger-backed projection behind the existing |
| 544 | runtime auth middleware: |
| 545 | |
| 546 | ```text |
| 547 | GET /v1/fleet/runs |
| 548 | GET /v1/fleet/runs/{run_id} |
| 549 | GET /v1/fleet/runs/{run_id}/workers |
| 550 | GET /v1/fleet/workers/{worker_id} |
| 551 | POST /v1/fleet/workers/{worker_id}/interrupt |
| 552 | POST /v1/fleet/workers/{worker_id}/restart |
| 553 | POST /v1/fleet/runs/{run_id}/stop |
| 554 | ``` |
| 555 | |
| 556 | Action endpoints call the same manager controls as the CLI and record their |
| 557 | decisions in the fleet ledger. |
| 558 | |
| 559 | ## Manager-Agent Runbook |
| 560 | |
| 561 | Manager agents should treat Fleet operations as typed, ledgered control-plane |
| 562 | work. Start with `codewhale fleet status`, then inspect one run or worker with |
| 563 | `codewhale fleet inspect <worker-id>`, `logs`, and `artifacts`. Use direct |
| 564 | reads of `.codewhale/fleet.jsonl`, host logs, or remote files only when the |
| 565 | typed CLI/API surface cannot provide the required evidence. |
| 566 | |
| 567 | Classify the worker before taking action: |
| 568 | |
| 569 | - `transient failure`: stale heartbeat, host timeout, interrupted transport, |
| 570 | retryable provider/network error, or an adapter status that can plausibly |
| 571 | recover without changing the task. |
| 572 | - `task failure`: the worker completed but produced an incorrect result, |
| 573 | domain failure, missing required artifact, or explicit task-level error. |
| 574 | - `verifier failure`: the worker result exists, but the scorer/verifier failed, |
| 575 | timed out, or disagrees with the receipt. |
| 576 | - `needs-human`: missing authority, secret request, destructive operation, |
| 577 | repeated restart exhaustion, ambiguous product decision, or conflicting |
| 578 | evidence that the manager cannot resolve from typed artifacts. |
| 579 | |
| 580 | Choose one typed action: |
| 581 | |
| 582 | - Restart a worker only when the failure is transient, retry budget remains, |
| 583 | the task is idempotent or retry-safe, and no permission or secret boundary is |
| 584 | involved: `codewhale fleet restart <worker-id>`. |
| 585 | - Interrupt or stop only when the current task is unsafe to continue or the |
| 586 | operator explicitly asks for cancellation: `codewhale fleet interrupt |
| 587 | <worker-id>` or `codewhale fleet stop --all`. |
| 588 | - Do not restart pure task failures by default; preserve artifacts and hand the |
| 589 | receipt to the task owner unless the task spec says retrying can produce new |
| 590 | evidence. |
| 591 | - For verifier failures, inspect scorer inputs and artifact refs first. If the |
| 592 | verifier cannot be corrected through typed fleet actions, escalate for human |
| 593 | review. |
| 594 | - For `needs-human`, draft an escalation instead of sending it unless alert |
| 595 | config explicitly authorizes sending. |
| 596 | |
| 597 | Safe Slack or PagerDuty draft: |
| 598 | |
| 599 | ```text |
| 600 | Codewhale fleet needs attention |
| 601 | Run: <run-id> |
| 602 | Worker: <worker-id> |
| 603 | Task: <task-id or unknown> |
| 604 | Classification: <transient failure | task failure | verifier failure | needs-human> |
| 605 | Reason: <one sentence, no secrets> |
| 606 | Latest typed evidence: codewhale fleet inspect <worker-id>; codewhale fleet artifacts <worker-id> |
| 607 | Safe log excerpt: <3 lines max or "see artifact <ref>"> |
| 608 | Requested decision: <restart approval | verifier review | task owner review | permission decision> |
| 609 | ``` |
| 610 | |
| 611 | Post-run summaries should include the run id, workers checked, classification, |
| 612 | typed action taken or drafted, expected ledger effect, artifact refs reviewed, |
| 613 | and next owner. Keep summaries bounded; link artifact refs instead of copying |
| 614 | full logs or transcripts. |
| 615 | |
| 616 | The bundled `fleet-manager` skill mirrors this runbook for manager agents. It |
| 617 | is a first-party system skill and should be discoverable through the normal |
| 618 | skill registry after system skills are installed or refreshed. |
| 619 | |
| 620 | ## Host Adapters |
| 621 | |
| 622 | The host adapter boundary supports local child processes and explicit SSH |
| 623 | workers. Adapters expose the same operations: start, read status, read bounded |
| 624 | logs, interrupt, restart, stop, and cleanup. |
| 625 | |
| 626 | Local workers run as child processes with stdin closed and stdout/stderr written |
| 627 | to bounded fleet host logs. They inherit only a small safe base environment |
| 628 | such as `PATH` and explicitly allowlisted variables. |
| 629 | |
| 630 | SSH workers run through the system `ssh` client with `BatchMode=yes` and a |
| 631 | bounded connect timeout. Remote environment variables are sent with OpenSSH |
| 632 | `SendEnv`; values are not embedded in the local ssh argv or fleet logs. |
| 633 | |
| 634 | Example SSH worker spec: |
| 635 | |
| 636 | ```json |
| 637 | { |
| 638 | "id": "builder-1", |
| 639 | "name": "Builder 1", |
| 640 | "host": { |
| 641 | "kind": "ssh", |
| 642 | "host": "builder.example.com", |
| 643 | "user": "codewhale", |
| 644 | "port": 22, |
| 645 | "identity": "~/.ssh/codewhale_fleet", |
| 646 | "working_directory": "/srv/codewhale/work", |
| 647 | "env_allowlist": ["CODEWHALE_PROFILE"], |
| 648 | "codewhale_binary": "/usr/local/bin/codewhale" |
| 649 | }, |
| 650 | "capabilities": ["local", "linux", "tests"], |
| 651 | "max_concurrent_tasks": 1 |
| 652 | } |
| 653 | ``` |
| 654 | |
| 655 | Defaults are intentionally conservative: |
| 656 | |
| 657 | - no hosted control plane or cloud provisioning is enabled; |
| 658 | - SSH requires an explicit host, working directory, and Codewhale binary path; |
| 659 | - secret-like environment names such as `TOKEN`, `SECRET`, `PASSWORD`, |
| 660 | `API_KEY`, and `PRIVATE_KEY` are rejected from adapter allowlists; |
| 661 | - secrets should remain in Codewhale config providers or remote host config, |
| 662 | not in task instructions, argv, or fleet logs. |
| 663 | |
| 664 | ## Security and Trust Boundaries |
| 665 | |
| 666 | Agent Fleet enforces a trust-level model that separates workers into four tiers. |
| 667 | The trust level determines what a worker can access (secrets, network, workspace |
| 668 | writes) and how it must prove its identity before being granted those privileges. |
| 669 | |
| 670 | ### Trust Levels |
| 671 | |
| 672 | | Level | Access | Requires | |
| 673 | |-------|--------|----------| |
| 674 | | `sandbox` | No network, no secrets, writes only to `.codewhale/fleet/` | Nothing — default for new workers | |
| 675 | | `local` | Workspace reads, gated writes, configured secrets | Local process (same uid) | |
| 676 | | `remote-verified` | Network access, bounded capability grants, configured secrets | SSH host-key verification or equivalent attestation | |
| 677 | | `operator` | Full access to all secrets, unrestricted writes, any action | Operator-owned machine | |
| 678 | |
| 679 | The default trust level is `sandbox`. Operators must explicitly raise trust for |
| 680 | SSH or container workers through the security policy. |
| 681 | |
| 682 | ### Security Policy |
| 683 | |
| 684 | A fleet run may carry an optional `security_policy` block that defines the |
| 685 | default trust level, which secrets workers may resolve, what capabilities are |
| 686 | granted, and a ceiling on the maximum trust level: |
| 687 | |
| 688 | ```json |
| 689 | { |
| 690 | "security_policy": { |
| 691 | "default_trust_level": "sandbox", |
| 692 | "allowed_secrets": [ |
| 693 | {"key": "GH_TOKEN", "source": "env"}, |
| 694 | {"key": "CODEWHALE_API_KEY", "source": "keyring"} |
| 695 | ], |
| 696 | "capability_grants": [ |
| 697 | { |
| 698 | "capability": "network", |
| 699 | "scope": "github.com", |
| 700 | "reason": "PR review needs GitHub API access" |
| 701 | } |
| 702 | ], |
| 703 | "max_trust_level": "remote_verified", |
| 704 | "require_identity_verification": true |
| 705 | } |
| 706 | } |
| 707 | ``` |
| 708 | |
| 709 | When a run has no explicit `security_policy`, workers inherit conservative |
| 710 | defaults: `sandbox` trust, no secrets, no capability grants, and no identity |
| 711 | verification requirement. |
| 712 | |
| 713 | ### Secret References |
| 714 | |
| 715 | Secrets are never stored as plaintext in task specs, alert configs, or worker |
| 716 | definitions. Instead, every secret is a `FleetSecretRef` — a key name plus an |
| 717 | optional source hint that tells the fleet manager where to resolve the value: |
| 718 | |
| 719 | ```json |
| 720 | {"key": "GH_TOKEN", "source": "env"} |
| 721 | ``` |
| 722 | |
| 723 | Supported sources: |
| 724 | - `"env"` — resolve from a process environment variable |
| 725 | - `"keyring"` — resolve from the OS keyring (macOS Keychain, Windows Credential Manager, Linux Secret Service) |
| 726 | - `"file"` — resolve from `~/.codewhale/secrets/` |
| 727 | - absent — try all sources in default order (store first, then env) |
| 728 | |
| 729 | Secret refs are redacted in logs and ledger entries: `<secret:env.GH_TOKEN>`. |
| 730 | |
| 731 | ### Worker Authentication |
| 732 | |
| 733 | Workers authenticate to the fleet manager using one of three methods: |
| 734 | |
| 735 | - **None** — local workers sharing the same uid (default) |
| 736 | - **SSH key** — with optional host-key fingerprint pinning and known-hosts |
| 737 | verification. The `host_key_fingerprint` field (SHA256:...) pins the expected |
| 738 | server key, preventing MITM attacks on first connection. |
| 739 | - **Token** — a bearer token resolved from a `FleetSecretRef`, useful for remote |
| 740 | workers behind a fleet proxy. |
| 741 | - **mTLS** — mutual TLS with a client certificate and a secret-backed private key. |
| 742 | |
| 743 | SSH workers should always set `host_key_fingerprint` in production: |
| 744 | |
| 745 | ```json |
| 746 | { |
| 747 | "id": "builder-1", |
| 748 | "name": "Builder 1", |
| 749 | "trust_level": "remote_verified", |
| 750 | "host": { |
| 751 | "kind": "ssh", |
| 752 | "host": "builder.example.com", |
| 753 | "user": "codewhale", |
| 754 | "port": 22, |
| 755 | "identity": "~/.ssh/codewhale_fleet", |
| 756 | "host_key_fingerprint": "SHA256:aLGqZo1M6c...", |
| 757 | "known_hosts": "~/.ssh/known_hosts", |
| 758 | "working_directory": "/srv/codewhale/work", |
| 759 | "env_allowlist": ["CODEWHALE_PROFILE"], |
| 760 | "codewhale_binary": "/usr/local/bin/codewhale" |
| 761 | }, |
| 762 | "capabilities": ["local", "linux", "tests"], |
| 763 | "max_concurrent_tasks": 1 |
| 764 | } |
| 765 | ``` |
| 766 | |
| 767 | ### Alert Channel Secrets |
| 768 | |
| 769 | Alert channels (Slack, generic webhook, PagerDuty) use `FleetAlertEndpoint` |
| 770 | instead of raw URLs. The webhook URL can be provided inline for non-sensitive |
| 771 | endpoints, or as a secret reference: |
| 772 | |
| 773 | ```json |
| 774 | { |
| 775 | "kind": "slack", |
| 776 | "webhook": { |
| 777 | "url_ref": {"key": "CODEWHALE_FLEET_SLACK_WEBHOOK", "source": "env"}, |
| 778 | "secret_ref": {"key": "CODEWHALE_FLEET_SLACK_SIGNING_SECRET", "source": "keyring"} |
| 779 | } |
| 780 | } |
| 781 | ``` |
| 782 | |
| 783 | The `secret_ref` field provides an optional HMAC secret for webhook payload |
| 784 | signing, never stored in plaintext. |
| 785 | |
| 786 | ### Config File |
| 787 | |
| 788 | The `[fleet]` table in `config.toml` sets global trust policy defaults: |
| 789 | |
| 790 | ```toml |
| 791 | [fleet] |
| 792 | default_trust_level = "sandbox" |
| 793 | require_identity_verification = true |
| 794 | max_trust_level = "operator" |
| 795 | |
| 796 | [fleet.exec] |
| 797 | # Recursion depth shares ONE axis with standalone sub-agents — a fleet worker |
| 798 | # IS a headless sub-agent. 0 blocks child agents (the root worker still runs); |
| 799 | # 3 is the default; explicit config clamps to the shared safety ceiling. |
| 800 | max_spawn_depth = 3 |
| 801 | ``` |
| 802 | |
| 803 | These defaults apply to fleet runs that don't carry their own `security_policy`. |
| 804 | Per-run policies always override the config defaults. |
| 805 | |
| 806 | ### Capability Grants |
| 807 | |
| 808 | Capability grants are additive, scoped permissions that authorize specific |
| 809 | actions. By default, workers get no grants (least privilege). Common grants: |
| 810 | |
| 811 | - `"network"` with scope `"github.com"` — allow outbound HTTP to GitHub |
| 812 | - `"git-push"` — allow `git push` to remotes |
| 813 | - `"provider-secrets"` — allow accessing provider API keys |
| 814 | - `"release"` — allow release-related operations (tagging, publishing) |
| 815 | - `"workspace-write"` with scope `"crates/tui/**"` — allow writes within a path |
| 816 | |
| 817 | ### Environment Sanitization |
| 818 | |
| 819 | The host adapter layer enforces environment sanitization at worker start: |
| 820 | |
| 821 | - Only `HOME`, `PATH`, and platform-specific vars (`SYSTEMROOT`, `COMSPEC`) are |
| 822 | injected into worker processes by default |
| 823 | - Environment allowlists reject any key containing `SECRET`, `TOKEN`, `PASSWORD`, |
| 824 | `PASSWD`, `API_KEY`, `CREDENTIAL`, or `PRIVATE_KEY` |
| 825 | - SSH workers only send explicitly allowlisted variables via OpenSSH `SendEnv` |
| 826 | - Secret values are never embedded in worker argv, task instructions, or fleet |
| 827 | logs — only secret refs appear, and they are always redacted |
| 828 |