| 1 | # Computer Use |
| 2 | |
| 3 | This is the Computer Use plugin included in Codewhale. The Engine embeds the |
| 4 | runtime bundle, discovers it through the existing plugin registry, and runs |
| 5 | only the copy the user has reviewed and enabled. Codewhale Apps uses that same |
| 6 | Engine inventory and approval flow. |
| 7 | |
| 8 | The bundle provides the current consolidated MCP tools for application and window observation, |
| 9 | accessibility actions, screenshots and zoom, keyboard and pointer input, |
| 10 | clipboard access, recording, and switching between registered computers. |
| 11 | Implementation exists for macOS, Windows, Linux and HarmonyOS target devices; |
| 12 | platform support still depends on the tools, OS grants and actual device |
| 13 | verification described by the upstream project. A source build is not a |
| 14 | published or certified release. |
| 15 | The bundled plugin is enabled on macOS only while Windows and Linux ports |
| 16 | are being qualified. Their raw input uses the shared desktop; they do not |
| 17 | yet provide equivalent background control or qualified native installers. |
| 18 | HarmonyOS and SSH also require separate device and workflow evidence. |
| 19 | |
| 20 | ## Included runtime |
| 21 | |
| 22 | macOS Codewhale builds carry the compiled native helper. Using the included |
| 23 | plugin needs neither a separate Computer Use app nor a compiler. It uses the |
| 24 | permission identity of its hosting Codewhale app or terminal. Accessibility |
| 25 | and Screen Recording grants remain controlled by the user in System Settings. |
| 26 | Use `request_access` to inspect readiness; a loaded plugin alone does not prove |
| 27 | its OS permissions work. |
| 28 | |
| 29 | When the standalone Computer Use helper is registered, it owns local input |
| 30 | even when Codewhale carries an embedded native helper. Version 0.12.0 keeps its |
| 31 | whale menu, permission setup, disposable background check and human |
| 32 | Pause/Stop controls, and retires the daemon when its native owner disappears. A registered helper that cannot start causes a clear |
| 33 | error; the client does not silently bypass its controls. Without a registered |
| 34 | standalone app, the included helper remains available under the host's |
| 35 | permission identity. |
| 36 | |
| 37 | The MCP server requires Node.js 20 or newer. Codewhale Apps packages its own |
| 38 | Node runtime; the CLI uses Node on PATH. Homebrew declares the dependency; |
| 39 | Cargo and direct binary users can install Node from <https://nodejs.org/>. |
| 40 | Linux also needs the appropriate X11 or Wayland utilities and AT-SPI bindings. |
| 41 | Windows uses PowerShell and UI Automation. Linux and Windows recording is |
| 42 | currently unavailable until recorder ownership and shutdown cleanup are built. |
| 43 | HarmonyOS targets require a connected device and hdc. |
| 44 | |
| 45 | Persistent holds and drags on Linux and Windows currently require the separate |
| 46 | session-aware Computer Use helper. Their direct bundled path refuses these |
| 47 | operations before sending input. macOS carries its native input owner in the |
| 48 | included bundle. Real Windows, Wayland and mixed-display validation is still |
| 49 | required before claiming equivalent platform readiness. |
| 50 | SSH and Docker use persistent session transports; real remote workflows still |
| 51 | need target-specific acceptance. The embedded Docker build context supports |
| 52 | first-use creation of a task-owned Linux desktop when a compatible daemon is available. |
| 53 | |
| 54 | ## Control and session ownership |
| 55 | |
| 56 | Select an application before sending input. On macOS, background selection |
| 57 | (`activate:false`) supports process-directed typing and accessibility actions. |
| 58 | It refuses gestures and keyboard shortcuts that would borrow the user's keyboard focus or move the shared pointer. Some Unicode and hosted-panel |
| 59 | typing also refuses rather than taking a focus lease. Explicit |
| 60 | foreground selection (`activate:true`) enables guarded foreground input |
| 61 | when the user has authorized exclusive desktop use. In both modes pointer input |
| 62 | goes to the bound app's window as window-routed events; the user's cursor is |
| 63 | never moved. Neither mode is an isolated computer. |
| 64 | Screenshots and zoom return actual image content to compatible vision models. |
| 65 | The nonactivating preview is on by default after binding; recording is explicit. |
| 66 | Application observations return a concise default summary; request full detail |
| 67 | when needed. Text-only models can use element roles, values and advertised |
| 68 | actions. On macOS, optional local OCR enriches the selected window observation |
| 69 | with text and raster bounds; it requires Screen Recording permission and does |
| 70 | not invent accessibility elements or actions. |
| 71 | The Engine permits one inline image up to 5 MiB per tool result; use a scoped |
| 72 | capture or zoom when a larger image receives an omission receipt. |
| 73 | |
| 74 | Each task owns its MCP connection and computer selection, observations and |
| 75 | held input. Sub-agents never receive Computer Use tools: only the task's own |
| 76 | agent operates the computer. Stopping control or closing the task releases that session's input. Stale |
| 77 | observations, unexpected foreground changes and unavailable capabilities fail |
| 78 | closed with a receipt; successful dispatch still needs application-state |
| 79 | verification. |
| 80 | |
| 81 | ## Development |
| 82 | |
| 83 | The exact upstream source revision is recorded beside this directory in |
| 84 | `computer-use.upstream-sha`. This tree contains the runtime and its tests; |
| 85 | standalone app installers and release tooling belong to the upstream project. |
| 86 | |
| 87 | Run `npm test` here for unit and protocol coverage. Those tests do not type or |
| 88 | click in the user's applications. `npm run smoke` is a separate legacy live |
| 89 | check: it captures and records the selected display, so run it only when that |
| 90 | capture is intended. The upstream parity suite contains scoped application |
| 91 | fixtures for interactive verification. |
| 92 | |
| 93 | On macOS, ordinary observations follow the selected background app. Field |
| 94 | focus, selection, context menus and scrolling use supported accessibility |
| 95 | operations; raw mouse gestures stop if the user changes foreground apps. |
| 96 | Arbitrary background dragging remains unavailable. Rebuild Core to include |
| 97 | the updated native helper; updating a separate marketplace checkout alone |
| 98 | does not update an already-installed Core binary. |
| 99 | |
| 100 | This embedded source matches canonical 9a261c4. Windows controlled-desktop |
| 101 | acceptance passed in upstream CI; signed Windows distribution, mixed-DPI/raw |
| 102 | input and continuous keyboard coexistence remain unqualified. Core discovery |
| 103 | and materialization tests do not constitute an installed model-driven trial. |
| 104 |