| 1 | # Quick reference |
| 2 | |
| 3 | Every tool takes an optional `computer` id (sticky switch). Every action |
| 4 | receipt is JSON: `ok`, plus what was sent. Verify effects by observing. |
| 5 | |
| 6 | Interface order per step: the host's own shell/files/APIs → `app_script` |
| 7 | → `browser` (CDP) → accessibility elements → pixels. Click only what has |
| 8 | no better interface. |
| 9 | |
| 10 | ## Observe |
| 11 | - `request_access` — permissions + capabilities; call once per session. |
| 12 | - `list_apps {all?}` — running apps (names, pids). Default: user-facing apps. |
| 13 | - `list_windows {app_ref?}` — windows with indices for `window_id`. |
| 14 | - `get_app_state {app_ref?, query?, role?, limit?, detail?}` — elements + |
| 15 | `state_id`. The targeting tree. |
| 16 | - `find_elements {state_id?, query?, role?}` — filter a cached observation. |
| 17 | - `wait_for {query|role, state, timeout?}` — poll until UI appears/disappears. |
| 18 | - `screenshot {app_ref?|region?|display?}` — raster for visual work. |
| 19 | - `zoom {region}` — magnify the last raster. |
| 20 | - `get_value {target}` — read an element's value. |
| 21 | - `cursor_position` — hardware pointer. |
| 22 | - `list_sessions` — who is driving this machine: live sessions with bound targets, modes, and held pointers. |
| 23 | - `clipboard {action:"read"}` — user clipboard text (ask before reading if unsure). |
| 24 | |
| 25 | ## Act |
| 26 | - `click {target, button?, clicks?}` — left (1–3 clicks), right, or middle. |
| 27 | - `type {text, target?, press_enter?}` — unicode-safe; verifies by read-back where possible. |
| 28 | - `key {text, repeat?|duration?}` — chords like `cmd+s`; `duration` holds the key. |
| 29 | - `set_value {target, value}` — semantic write with read-back. |
| 30 | - `select_text {target, text_range?}` · `focus {target}` · `perform_action {target, action}` |
| 31 | - `scroll {target, direction, amount?}` · `left_click_drag {from_target, to}` |
| 32 | - `invoke_menu {path}` — app menu items through accessibility (exact for app-level commands like New/Save/Quit; see the close recipe for windows). |
| 33 | - `pointer {action, target?}` — move/down/up primitives (foreground/shared only). |
| 34 | - `app_script {script, language?, timeout?}` — macOS local only: AppleScript |
| 35 | (default) or JXA through osascript. `result` is stdout; refusals are |
| 36 | `script_error`, `script_timeout`, `automation_denied` (-1743 consent), |
| 37 | `script_refused` (shell escapes, ObjC bridge, terminal apps, or an app not |
| 38 | named with a literal) and `unsupported_on_transport` on ssh/docker/hdc. |
| 39 | Every app the script names needs consent like any other target. |
| 40 | |
| 41 | ## Apps & computers |
| 42 | - `open_application {name|bundle_id|pid, activate?}` — bind the input target; `app_not_found` when the selector resolves nowhere. |
| 43 | - `list_apps {installed:true}` — the installed catalog (openable apps, running or not, one subdirectory deep) instead of the running list. |
| 44 | - `kill_app {name|bundle_id|pid, force?}` — quit an app; refuses an ambiguous name match (pass pid); never the helper itself. |
| 45 | - `set_window_frame {app_ref?, window_id, frame:{x,y,w,h}}` — move/resize one window; readbacks report what the app actually did (`verified`, `ax_errors`). |
| 46 | - `preview {enabled}` — floating panel: captured window + agent/user cursors; live while bound. |
| 47 | - `computer {action, id?}` — list / switch / register / **spawn** / remove. |
| 48 | `spawn {id, transport:"docker", image?}` provisions a task-owned disposable |
| 49 | Linux desktop (registered `owned:true`, becomes active) — the default |
| 50 | workspace for anything that does not need the user's own session. `remove` |
| 51 | or session end destroys it. `local` stays for work in the user's session. |
| 52 | - `recording {action, …}` — start / stop / status / list (opt-in screen recordings). |
| 53 | |
| 54 | ## Browser (CDP) |
| 55 | - `browser {action:"start", url?}` — self-owned Chromium profile + this session's tab; the user's own browser is never touched. |
| 56 | - `browser {action:"navigate", url}` — http(s) or about:blank; waits for load (`verified`). |
| 57 | - `browser {action:"click", selector | point}` — CSS selector's box center, or page-viewport pixels from a browser screenshot. |
| 58 | - `browser {action:"type", text, selector?, enter?}` — inserts text (unicode), optional focus selector and Enter. |
| 59 | - `browser {action:"screenshot", full?}` — page PNG (inline image); viewport is the click-point space. |
| 60 | - `browser {action:"status"}` · `browser {action:"stop"}` — tabs/active tab; close this session's tab (last one out closes the browser). |
| 61 | |
| 62 | ## Session |
| 63 | - `consent {action:"status"|"allow"|"deny"|"revoke", app?|scope?}` — the |
| 64 | per-app decision ledger on `local`. First contact with an app refuses |
| 65 | `consent_required`; record the user's answer (`remember:true` persists). |
| 66 | `scope:"foreground"` is the separate shared-pointer decision |
| 67 | `open_application activate:true` needs. A denied app fails `app_denied` |
| 68 | under every spelling; only the user can revoke it. |
| 69 | `consent {action:"allow", confirm:"<token>"}` records the user's approval |
| 70 | of one exact pay/buy/send/transfer/delete call that refused |
| 71 | `confirmation_required` — only after they approved it. |
| 72 | - `list_sessions` — live sessions on this machine (content-free) and the user's control mode. |
| 73 | - `trajectory {action:"start"|"stop"|"status"|"replay", id?, dry_run?}` — record this session's tool calls to a local, owner-only JSONL (entered text redacted; those steps do not replay); replay re-enters the normal pipeline and stops at the first refusal. |
| 74 | - `stop_computer_control {reason?}` — kill switch; input for this session ends. |
| 75 | - Capability grant (host config): `CODEWHALE_CU_GRANT="read-only"` or a tool list — the session can never see or call beyond it (`not_granted`). |
| 76 | |
| 77 | Receipt fields worth reading: `yield_ms` is how long an action waited for a |
| 78 | gap in the user's hardware input before taking a shared surface; a deny |
| 79 | never carries `action_sent`. |
| 80 | |
| 81 | ## Recipes |
| 82 | |
| 83 | Type into the document body: |
| 84 | 1. `open_application {name:"TextEdit", activate:false}` |
| 85 | 2. `get_app_state {role:"AXTextArea"}` → index |
| 86 | 3. `type {target:{type:"element",index}, text:"…"}` |
| 87 | 4. `get_value {target:{type:"element",index}}` → confirm |
| 88 | |
| 89 | Close a window without borrowing focus: |
| 90 | 1. Press the window's close-button element (`click` on the window's |
| 91 | `AXButton`), or use an available `invoke_menu` close action. Modified keys |
| 92 | refuse in background mode because they need keyboard focus. |
| 93 | |
| 94 | Fill and submit a web form (CDP, no pixels): |
| 95 | 1. `browser {action:"start", url:"https://…"}` |
| 96 | 2. `browser {action:"type", selector:"#email", text:"…"}` |
| 97 | 3. `browser {action:"click", selector:"button[type=submit]"}` |
| 98 | 4. Verify with `browser {action:"status"}` (url/title) or a fresh |
| 99 | `browser {action:"screenshot"}` — never assume the click landed. |
| 100 | |
| 101 | Record and re-verify a session: |
| 102 | 1. `trajectory {action:"start"}` |
| 103 | 2. …do the work… |
| 104 | 3. `trajectory {action:"stop"}` → file + turns |
| 105 | 4. `trajectory {action:"replay", id, dry_run:true}` to review, then replay |
| 106 | without `dry_run` to re-run through the same gates. |
| 107 | 2. `list_windows` → window gone |
| 108 | |
| 109 | > `invoke_menu` is exact for app-level commands (New, Save, Quit). |
| 110 | > Window-targeted items like Close can validate against a key window that a |
| 111 | > background app does not have and legitimately no-op — verify the effect |
| 112 | > before retrying or reporting. |
| 113 | |
| 114 | Move between apps mid-task: `open_application` retires the previous app's |
| 115 | indices; always `get_app_state` after switching. |
| 116 | |
| 117 | > The per-action wire names (`left_click`, `read_clipboard`, `recording_start`, |
| 118 | > `computer_list`, `hold_key`, …) remain callable as aliases; `tools/list` |
| 119 | > advertises the merged set above. |
| 120 |