| 1 | # Desktop browser |
| 2 | |
| 3 | [简体中文](DESKTOP_BROWSER.zh-CN.md) |
| 4 | |
| 5 | The desktop browser is a native Chromium surface inside the Reasonix window |
| 6 | that the user and the agent operate together. Websites render in Electron |
| 7 | `WebContentsView`s owned by the shell; every agent capability goes through the |
| 8 | Go desktop service so that local and remote agents, approvals, cancellation, |
| 9 | evidence and operation records share one implementation. This document is the |
| 10 | contract between the browser panel, the shell's surface manager, the Go |
| 11 | `BrowserExecutor` and the tools the agent sees. Sessions with no shell behind |
| 12 | them get the same tools from the [CDP backend](BROWSER_CDP.md). |
| 13 | |
| 14 | ```text |
| 15 | agent tool call ─▶ Go BrowserExecutor ─▶ ledger.reserve ─▶ host/browser.* ─▶ WebContentsView |
| 16 | ▲ │ │ |
| 17 | └── result/evidence ┴──────────── ledger.settle ◀─────────┘ |
| 18 | user input on the page ─▶ guest preload ─▶ shell: epoch++ ─▶ desktop/event browser:takeover |
| 19 | ``` |
| 20 | |
| 21 | ## Surfaces and trust |
| 22 | |
| 23 | - The application window is trusted. Website views are not: sandbox on, |
| 24 | context isolation on, no Node integration, no application preload, no |
| 25 | `reasonix://` access. Their only preload observes trusted user input to |
| 26 | request a take-over and exposes nothing to the page. |
| 27 | - Partitions: `persist:browser` is the shared login partition for one |
| 28 | Reasonix data home; `temp:<id>` partitions are in-memory and discarded when |
| 29 | their last tab closes. Remote Serve windows and MCP App frames use their |
| 30 | own partitions and never share the browser partition. |
| 31 | - The `BrowserSurfaceManager` in the shell owns creation, visibility, bounds, |
| 32 | focus and destruction. The React panel submits a layout rectangle; the shell |
| 33 | validates it against the window and applies it. Any application overlay |
| 34 | (dialogs, menus, command palette) sets a single overlay state that hides |
| 35 | every native website view so a page can never paint over the app. |
| 36 | - One task owns its tabs. A tab carries `{tabID, taskID, sessionID, epoch, |
| 37 | partition}`; switching the visible tab never retargets a running plan. |
| 38 | |
| 39 | ## Panel |
| 40 | |
| 41 | The right workspace gains a browser panel: tab strip per task, address bar, |
| 42 | back/forward, reload, zoom, load errors with retry, download list, DevTools |
| 43 | toggle, responsive viewports, element selection, diagnostics and recording. |
| 44 | Restored tabs keep task/session ownership, URL/title/order and viewport preferences; |
| 45 | no grants, form state, DOM refs or replayable submissions are persisted. The shell |
| 46 | stores metadata in `browser-tabs-v1.json` under Electron userData. Tabs restore as |
| 47 | placeholders and load on explicit access. Temporary tabs are excluded. Local HTML |
| 48 | previews retain their original file reference and require fresh authorization; |
| 49 | their expiring preview URL is never persisted. The existing operation ledger is |
| 50 | unchanged. Each window allows at most 32 logical tabs, without automatic eviction. |
| 51 | |
| 52 | Local `.html` and `.htm` files run in this panel by default. Chat file links, |
| 53 | present cards, the file tree and on-demand agent preview share one path: the |
| 54 | desktop re-authorises the original file source, creates a random-token loopback |
| 55 | HTTP URL, and binds it to a tab owned by the current task and session. **View |
| 56 | source** still opens the file panel, while **Open in external browser** is an |
| 57 | explicit user action. Reload re-authorises and replaces the token; closing the |
| 58 | tab, destroying the session or navigating away releases the binding. A taken |
| 59 | over tab cannot be refreshed or duplicated by the agent. An explicit user |
| 60 | reload updates the page while leaving the tab in `human` mode. |
| 61 | |
| 62 | This file path serves single files and resources allowed by the existing |
| 63 | static-preview policy. Projects that need a module build, client routing or a |
| 64 | local API still use the terminal to start or reuse the project's declared |
| 65 | development server, then open its confirmed address with `browser_open`. SSH |
| 66 | files retain their existing file preview in the first release. |
| 67 | |
| 68 | ## Browser control settings |
| 69 | |
| 70 | The Settings Centre page "Browser control" owns the switches below, all stored |
| 71 | in `browser-control.json` inside the shell's userData profile (`desktop-shell/`). |
| 72 | |
| 73 | - **Built-in browser control** (`controlEnabled`, default on). The shell pushes |
| 74 | it to Go with `desktop/browserControl`, and Go reads it when it builds a |
| 75 | session: a new session registers no browser tool at all, while a running |
| 76 | session keeps the tool set it started with. |
| 77 | - **Ignore certificate errors** (`ignoreCertificateErrors`, default off). |
| 78 | Relaxes `setCertificateVerifyProc` for guest sessions only. It is applied when |
| 79 | the guest-view factory prepares a partition and re-applied to every live guest |
| 80 | session when the switch changes, so no restart is needed. |
| 81 | - **Clear built-in browser cache** clears the HTTP cache plus Cache Storage, |
| 82 | Service Workers and the shader cache of `persist:browser`, keeping cookies and |
| 83 | local site data. |
| 84 | - **Clear all browser data** additionally drops every storage type, which signs |
| 85 | every site in the built-in browser out. In-memory `temp:<id>` partitions are |
| 86 | unaffected. |
| 87 | - **Import Chrome sign-in state** reads the newest Chrome profile's `Cookies` |
| 88 | database, decrypts each value (macOS: the login keychain's `Chrome Safe |
| 89 | Storage` secret; Linux: the well-known `peanuts` password; Windows: the DPAPI |
| 90 | master key from `Local State`) and writes it into `persist:browser`. Expired |
| 91 | cookies, App-Bound (`v20`) values and rows that fail to decrypt are counted as |
| 92 | skipped. Passwords are never read. |
| 93 | |
| 94 | ## Agent capabilities |
| 95 | |
| 96 | Tools are registered through the existing capability registry as one |
| 97 | `browser` capability with these operations. Every write goes through the |
| 98 | normal approval policy (`ask`, `allow`, `deny`), the normal cancellation |
| 99 | context and the evidence trajectory; there is no separate browser approval |
| 100 | system. |
| 101 | |
| 102 | | Tool | Reads/Writes | Purpose | |
| 103 | | --- | --- | --- | |
| 104 | | `browser_tabs` | read | list the task's tabs with URL, title, loading state | |
| 105 | | `browser_open` | write | open a tab (shared or temporary partition) at a URL | |
| 106 | | `browser_preview` | write | authorise a local task file and run or refresh it in a task-bound tab | |
| 107 | | `browser_navigate` | write | navigate the bound tab (URL, back, forward, reload) | |
| 108 | | `browser_snapshot` | read | structural snapshot with element references | |
| 109 | | `browser_screenshot` | read | PNG of the viewport or an element, returned as an image | |
| 110 | | `browser_click` | write | click a referenced element (trusted mouse events at its centre) | |
| 111 | | `browser_type` | write | type text into a referenced element with trusted key events; optional submit | |
| 112 | | `browser_press` | write | press a key or chord | |
| 113 | | `browser_scroll` | write | scroll the viewport or an element | |
| 114 | | `browser_select` | write | choose options in a select | |
| 115 | | `browser_upload` | write | attach task files to a file input | |
| 116 | | `browser_download` | read | wait for or list downloads of the tab | |
| 117 | | `browser_close` | write | close a tab | |
| 118 | |
| 119 | `browser_preview` is discoverable through `use_capability` only when a local |
| 120 | desktop executor provides the shared file-preview service. It does not enter |
| 121 | the always-on tool schema or inject URLs, ports or tab state into the system |
| 122 | prompt. |
| 123 | |
| 124 | Enhanced hosts additionally expose `browser_query`, `browser_wait`, |
| 125 | `browser_viewport`, `browser_pointer`, `browser_diagnostics`, and `browser_record` |
| 126 | through the on-demand inventory. They use an optional executor interface and |
| 127 | per-capability negotiation; an older host explicitly reports unsupported. |
| 128 | Existing tool descriptions/schemas and the permanent provider prefix are unchanged. |
| 129 | Query uses current document refs, reports ambiguous matches, and can scope to a |
| 130 | container or to the frame containing an existing ref. Wait is bounded (default |
| 131 | 3 seconds, maximum 8), cancellable, and does not require network idle. |
| 132 | Coordinate actions require a current screenshot observation token. Unknown |
| 133 | write outcomes are never replayed. |
| 134 | |
| 135 | The page runtime is built independently with the qualified Playwright 1.62.1 |
| 136 | injection resource; it is not serialized from main-process functions. Snapshot |
| 137 | and screenshot replies optionally carry document/viewport identity and timing. |
| 138 | PNG data must pass generation-side and Go-side decoding before model delivery. |
| 139 | Background capture currently requires macOS; unqualified platforms return a |
| 140 | show-page-and-retry error. Recording captures the existing page without audio, |
| 141 | defaults to 20 seconds, and is limited to 90 seconds / 64 MiB. Only a completed, |
| 142 | decoded WebM is published. The View menu and Ctrl/Command+Shift+R stop recording |
| 143 | even when the panel is closed. |
| 144 | |
| 145 | See the [implementation and acceptance record](BROWSER_RUNTIME_UPGRADE_ACCEPTANCE.md) |
| 146 | for evidence, platform gates and outstanding installed-package verification. |
| 147 | |
| 148 | Snapshot format: an accessibility-style tree (`role "name" [state] ref=e12`) |
| 149 | produced in an isolated world of the main frame and each reachable frame. |
| 150 | References are bound to `{tabID, frameID, documentVersion}`; a navigation, |
| 151 | page replacement or take-over invalidates every earlier reference, and an |
| 152 | action with a stale reference returns `not_executed: stale reference` rather |
| 153 | than guessing. Inputs and clicks are dispatched as trusted input events |
| 154 | through the shell, never by assigning element values, so React-controlled |
| 155 | inputs, custom widgets and dynamic pages behave as they would for a user. |
| 156 | |
| 157 | Screenshots and downloads never travel through control frames: the shell |
| 158 | writes them into the task's temporary directory that Go names in the request |
| 159 | and returns the path; Go turns the file into an image or file result through |
| 160 | the existing channels. Uploads read only files the task owns; remote tasks |
| 161 | stage files through the existing SFTP transfer and the task temporary |
| 162 | directory, so a remote path is never treated as a local one. |
| 163 | |
| 164 | ## Ownership, take-over and unknown writes |
| 165 | |
| 166 | - A grant `{sessionID, taskID, runtimeGeneration, tabIDs, expiresAt}` is |
| 167 | minted by Go when the task starts using the browser and revoked when the |
| 168 | service restarts, the session changes, the connection generation changes or |
| 169 | the task ends. The shell rejects `host/browser.*` calls whose grant is not |
| 170 | current. |
| 171 | - The browser toolbar's **Take over** button immediately switches the tab to |
| 172 | `human` mode and increments its epoch through trusted application IPC. |
| 173 | It works even during the 750 ms window that suppresses echoed agent input; |
| 174 | automatic keyboard, mouse and touch detection is best effort during that |
| 175 | window. Pending and queued actions for the old epoch are cancelled and Go |
| 176 | receives `browser:takeover`. **Resume** hands control back to the agent with |
| 177 | another epoch change, requiring a fresh page read. Login pages, captchas and passkeys are always a user |
| 178 | hand-over: while the tab is in `human` mode the agent cannot read or act on |
| 179 | it. |
| 180 | - Every write reserves an operation `{operationID, sessionID, generation, |
| 181 | tabID, epoch, documentToken, action, digest}` in the ledger before the |
| 182 | shell executes it. The shell reports `executed` or `not_executed` with a |
| 183 | reason; a lost reply, a crash or a service restart leaves the operation |
| 184 | `unknown`. Unknown operations are shown to the user and are never replayed |
| 185 | automatically; a reused `operationID` is rejected forever. |
| 186 | - Renderer crash of a website view cancels only that tab's actions and |
| 187 | reloads the last safe URL in `human` mode. Application renderer crash |
| 188 | pauses all browser actions until the UI re-attaches. |
| 189 | |
| 190 | ## Host calls |
| 191 | |
| 192 | | Method | Purpose | |
| 193 | | --- | --- | |
| 194 | | `host/browser.grant` `revoke` | install or revoke a grant | |
| 195 | | `host/browser.tabs.list` `open` `close` `activate` `navigate` | tab lifecycle bound to a grant | |
| 196 | | `host/browser.snapshot` | structural snapshot for a tab, returns `documentToken` | |
| 197 | | `host/browser.act` | one reserved action; returns `{executed, reason, documentToken}` | |
| 198 | | `host/browser.screenshot` | capture to a task-owned file path | |
| 199 | | `host/browser.downloads` | list or wait for downloads of a tab | |
| 200 | | `host/browser.layout` | apply the panel rectangle and overlay state | |
| 201 | |
| 202 | Events from the shell: `browser:tabs` (tab list changes), `browser:takeover` |
| 203 | (`{tabID, epoch, reason}`), `browser:download` (progress), `browser:crash`. |
| 204 | |
| 205 | ## Remote agents |
| 206 | |
| 207 | A remote Reasonix agent reaches the local browser through the existing SSH |
| 208 | connection and forward manager as a restricted host RPC carrying the same |
| 209 | `BrowserExecutor` contract. Grants are bound to the remote connection |
| 210 | generation, session and task; disconnect, reconnect or session switch revokes |
| 211 | them. Browser grants and provider-proxy credentials are separate; no shared |
| 212 | token. Older remote Serve builds negotiate capabilities and simply do not |
| 213 | advertise the browser, keeping every existing remote feature. |
| 214 | |
| 215 | The wire shape is one loopback HTTP broker per desktop. The bootstrap of a |
| 216 | fresh Serve injects `REASONIX_BROWSER_BROKER` / `REASONIX_BROWSER_TOKEN` |
| 217 | (process environment only) pointing at the reverse-forwarded broker; a reused |
| 218 | Serve is re-pointed through `POST /browser/broker` after the desktop rotates |
| 219 | the route. The broker mints one random bearer token per host connection |
| 220 | generation — registering a new generation replaces the host's old token — and |
| 221 | authenticates before dispatching to `browser.Executor` over |
| 222 | `/v1/browser/<method>`. Every request carries `X-Reasonix-Browser-Session`; |
| 223 | the broker resolves it to the one desktop tab that shows that session and |
| 224 | refuses anything else with `no_grant`. Screenshots and downloads the shell |
| 225 | writes on the desktop are staged onto the remote host through the existing |
| 226 | SFTP channel into a per-workspace scratch directory |
| 227 | (`~/.reasonix/browser-relay/<workspace>/`), so the serve's tools only ever |
| 228 | read paths local to them. A Serve started with a broker advertises `browser` |
| 229 | in the `X-Reasonix-Serve-Capabilities` header of the `/auth/token` handshake. |
| 230 | |
| 231 | ## Acceptance |
| 232 | |
| 233 | ### Right panel sizing |
| 234 | |
| 235 | Without an existing width preference, the first opening targets 45% of the |
| 236 | window width. Normal split layout uses a 300px minimum and a 70% window cap, |
| 237 | protects 400px for chat, and falls back to an overlay when there is insufficient |
| 238 | room. Pointer and keyboard resizing retain the existing local preference |
| 239 | format across closing, reopening, and restarting. Responsive compression never |
| 240 | writes back to that preference. |
| 241 | |
| 242 | Below 1024px the navigation sidebar collapses automatically, with a temporary |
| 243 | manual expansion override cleared on crossing the breakpoint. Wide-window |
| 244 | preferences are preserved. Below 768px the right panel fills the app content |
| 245 | area while keeping window chrome and a panel-close action reachable. Closing |
| 246 | returns to chat; widening restores the split. Files, Overview, and Browser use |
| 247 | the same rules. HTML reflow still depends on the document's CSS; the panel does |
| 248 | not scale or rewrite fixed-width pages. |
| 249 | |
| 250 | Run `pnpm exec tsx src/__tests__/responsive-dock.test.ts` and |
| 251 | `pnpm test:dock-responsive-browser`. The browser regression uses actual app |
| 252 | components with mock sessions to verify geometry, dragging, reopening, and |
| 253 | reload restoration. Add `--electron` to `node bench/responsive-dock.mjs` to run |
| 254 | the same checks against actual native window resizing with an isolated profile. |
| 255 | These geometry checks do not validate agent-driven HTML page execution. |
| 256 | |
| 257 | Iframes, dynamic DOM, controlled inputs, popups, upload and download, |
| 258 | navigation history, temporary partitions, shared and isolated logins; |
| 259 | take-over before approval, after approval before dispatch, lost receipt after |
| 260 | execution, restart after crash, duplicate operation IDs; remote SSH drop, |
| 261 | generation change, stale grant, cross-session misrouting. The real tasks in |
| 262 | [the migration record](DESKTOP_SHELL_MIGRATION.md#acceptance-gates) close |
| 263 | the phase. |
| 264 |