| 1 | # Conformance fixtures |
| 2 | |
| 3 | The gate for the Rust → TypeScript edge migration (`CURRENT_DECISIONS.md` §26; |
| 4 | TS-EXTENSION-HOST-DESIGN §9.3). A subsystem that moves to the extension host — |
| 5 | MCP client, hook orchestration, script tools, tool adapters, slash commands — |
| 6 | flips its flag only after it reproduces these goldens on these fixtures. The |
| 7 | goldens are what the Rust implementation does today; they are not a spec |
| 8 | written in the abstract. |
| 9 | |
| 10 | Everything here is language-neutral: JSON, JSONL, raw SSE bytes, and POSIX |
| 11 | shell. No fixture was captured from a real provider or contains a secret. |
| 12 | |
| 13 | The Rust runner is `crates/tui/src/conformance/` (one golden test per family, |
| 14 | plus hardening controls): |
| 15 | |
| 16 | ```sh |
| 17 | cargo test -p codewhale-tui --lib -- conformance:: |
| 18 | # Re-record after an intended change, then review the diff like source: |
| 19 | CODEWHALE_CONFORMANCE_UPDATE=1 cargo test -p codewhale-tui --lib -- conformance:: |
| 20 | ``` |
| 21 | |
| 22 | Default is compare-only. Update mode rewrites drifted goldens and refuses to |
| 23 | run when `CI` is set. A golden change is a behavior change and is reviewed as |
| 24 | one. Harness deadlines, incomplete turns, broken invariants, and empty cases |
| 25 | are failures in both modes and cannot be recorded as goldens. Provider timeout |
| 26 | errors are legitimate observable outcomes; a harness timing out is missing |
| 27 | evidence. |
| 28 | |
| 29 | Platforms: `sse` and `mcp` run everywhere. `events`, `prompt` and `hooks` |
| 30 | goldens were recorded on Unix and compile only there — hook fixtures are POSIX |
| 31 | shell, and a Windows turn differs in shell and path facts no golden covers |
| 32 | yet. Recording Windows goldens is open work, not an implied capability. |
| 33 | |
| 34 | Every `events` and `prompt` case declares `recorded_platform`: the execution |
| 35 | boundary its goldens were recorded under (`sandbox_enforcement`: `local_os`, |
| 36 | `unavailable` or `external_backend`; `no_new_privs_active`: `null` where the |
| 37 | flag does not exist, otherwise a bool). The Engine names this posture to the |
| 38 | model in `<turn_meta>`, so the harness replays the recorded facts instead of |
| 39 | probing the runner. Without that, a macOS-recorded golden fails on a Linux |
| 40 | runner for a fact about the machine, not the Engine. A case that omits the |
| 41 | field fails loud. The same record carries `os` and `shell` (`macos`, `/bin/bash`: the shell |
| 42 | dispatcher names the recording process's `$SHELL` path verbatim), |
| 43 | which the `## Environment` block renders into the frozen prompt prefix and so |
| 44 | into `prefix_cache_change`'s prefix hash; the harness replays those too, and |
| 45 | `host_tools`: the optional tools whose backend the recording machine had |
| 46 | (`code_execution`, `js_execution`, `pandoc_convert`, `image_ocr`), which |
| 47 | change the registry the `tool_request_snapshot` counts. |
| 48 | Production always probes the host. The per-platform label |
| 49 | text is owned and tested in `sandbox::policy`. All current cases were |
| 50 | recorded on macOS. A Linux- or Windows-recorded golden is separate evidence. |
| 51 | |
| 52 | ## Families |
| 53 | |
| 54 | Each case is `<family>/<name>.case.json` (input, hand-written) plus one or |
| 55 | more `<name>.golden.*` files (output, recorded). |
| 56 | |
| 57 | | family | input | production path | proves | |
| 58 | |---|---|---|---| |
| 59 | | `sse/` | `<name>.sse` recorded bytes + route | `CodewhaleClient::create_message_stream` via a loopback HTTP server | wire adapter: bytes → normalized stream events, incl. tool-call deltas, reasoning, usage, mid-stream errors, truncation | |
| 60 | | `events/` | scripted provider + host driver | `Engine::run` → `Engine::run_turn` | one turn → protocol `EventMsg` sequence + provider request count + workspace side effects | |
| 61 | | `mcp/` | scripted MCP server transcript + host steps | `McpPool` behind `Engine::execute_mcp_tool_with_pool` | catalog normalization and call results for one dispatch | |
| 62 | | `prompt/` | session config + workspace files | first `MessageRequest` of a real turn | model-visible prefix bytes (system prompt + tool catalog) | |
| 63 | | `hooks/` | user hook config + one tool call | `run_tool_call_before_hooks` / `HookExecutor::execute` | hook verdict fold, env contract, schema-1 stdin | |
| 64 | |
| 65 | The host protocol corpus that Rust serde and the TypeScript host both parse |
| 66 | already lives in `../extension_host/protocol/`; it is not duplicated here. |
| 67 | |
| 68 | ## Normalized formats |
| 69 | |
| 70 | ### Stream events (`sse` goldens, `events` provider scripts) |
| 71 | |
| 72 | One JSON object per event in exactly the serde shape of |
| 73 | `codewhale_models::StreamEvent` — the Anthropic Messages event vocabulary: |
| 74 | `message_start`, `content_block_start` (`text` / `thinking` / `tool_use` / |
| 75 | `server_tool_use`), `content_block_delta` (`text_delta` / `thinking_delta` / |
| 76 | `input_json_delta` / `signature_delta` / `reasoning_state_delta`), |
| 77 | `content_block_stop`, `message_delta` (stop reason + usage), `message_stop`, |
| 78 | `ping`, `error`, `tool_projection_warning`. The Rust runner proves every golden |
| 79 | event line deserializes into `StreamEvent` and serializes back unchanged, so an |
| 80 | `sse` golden can be pasted into an `events` script. |
| 81 | |
| 82 | An `sse` golden's first line is `{"request": [{"method", "path"}]}` — the |
| 83 | request the adapter sent. A stream that errors yields |
| 84 | `{"type": "stream_failure", "detail": …}`; a request that fails before a |
| 85 | stream yields `{"type": "open_failure", "detail": …}`. |
| 86 | |
| 87 | ### Turn events (`events` goldens) |
| 88 | |
| 89 | Each line is `codewhale_protocol::EventMsg` as serialized (`"event"` tag), with: |
| 90 | |
| 91 | - the per-session `thread_id` / `session_id` envelope removed; |
| 92 | - `tool_call_heartbeat` dropped (a liveness pulse, timing-dependent); |
| 93 | - UUIDs → `<uuid:N>` numbered by first appearance; RFC 3339 timestamps → |
| 94 | `<timestamp>`; the local date `<turn_meta>` states → `<today>`; `created_at`, `duration_ms`, `first_token_ms`, `request_ms`, |
| 95 | `elapsed_ms`, `pinned_combined_hash` → `"<masked>"`; temp paths → |
| 96 | `<WORKSPACE>` / `<HOME>` / `<TMP>`; |
| 97 | - `tool_catalog` and `system_prompt` bodies → `"<pinned by the prompt family>"`; |
| 98 | - an uninterrupted run of completion events put in a canonical order, |
| 99 | because parallel completions race and two tools' pairs can interleave: |
| 100 | `operation_activity_completed` observations by the established `span_id`, |
| 101 | then `tool_call_complete` events by `tool_call_id`; outcomes, span |
| 102 | relationships, and event counts stay exact, and no error or other event |
| 103 | is crossed. The activity events after a parallel batch's `Executing N ... |
| 104 | parallel chunk(s)` status are ordered the same way with starts first (a fast |
| 105 | tool can finish before its sibling starts), only when the run is causally |
| 106 | valid: every span starts once and completes once after its start, and a |
| 107 | tool completes after its own activity; |
| 108 | - keys sorted. |
| 109 | |
| 110 | The last line is `{"harness_summary": {model_requests, non_streaming_requests, |
| 111 | workspace_after}}`: how many provider requests the turn made and every file |
| 112 | left in the workspace (path → sha256 prefix). `invariants` in a case are |
| 113 | checked independently of the golden (`workspace_file_absent`, |
| 114 | `workspace_file_present`, `max_model_requests`). |
| 115 | |
| 116 | `events/provider_error_after_tool_call` requires the C02-05 (#6561) authority: |
| 117 | a failed response executes no collected tool call and issues no retry. Its |
| 118 | invariants are checked before recording, with no exemption for known defects. |
| 119 | |
| 120 | ### MCP transcripts (`mcp` goldens) |
| 121 | |
| 122 | The case's `server` object scripts a Streamable HTTP MCP server: one answer |
| 123 | per list method (`initialize`, `tools/list`, `resources/list`, |
| 124 | `resources/templates/list`, `prompts/list`) and, for `tools/call`, |
| 125 | `resources/read` and `prompts/get`, a list of `{"match": {…params}, …}` |
| 126 | entries answering with `result`, `error`, `progress` (notifications sent as |
| 127 | SSE before the result), or `hold` (never answer until the client goes away). |
| 128 | Notifications get `202`, a `GET` gets `405`, unknown methods get `-32601`. |
| 129 | |
| 130 | `steps` are what a host does: `catalog`, or `call` a model-facing tool name |
| 131 | with `input` (optionally `"cancel": "after_server_holds"`). The golden records |
| 132 | `boot`, the catalog (`codewhale_models::Tool` list), each call as |
| 133 | `{"ok": {success, content, metadata, content_blocks}}` (JSON content parsed) or |
| 134 | `{"err": {kind, detail}}`, and `server_received`: every `tools/call`, |
| 135 | `resources/read` and `prompts/get` the server actually saw. The server URL is |
| 136 | masked. The golden does not name the dispatch: every dispatch must produce the |
| 137 | same file. |
| 138 | |
| 139 | ### Prompt bytes (`prompt` goldens) |
| 140 | |
| 141 | `<name>.golden.json` holds sha256 of the system prompt JSON, of the tool |
| 142 | catalog JSON, and of both (`prefix_sha256`), with sizes and tool names in |
| 143 | catalog order; `.system.golden.txt` and `.tools.golden.json` are the readable |
| 144 | bodies. Bytes are hashed as serialized — key order is part of the cached |
| 145 | prefix. Masks: temp paths, and the environment block's `- platform:` and |
| 146 | `- shell:` lines. Tools admitted only after probing the host for a binary |
| 147 | (`code_execution`, `image_ocr`, `js_execution`, `pandoc_convert`) are removed |
| 148 | from the pinned catalog and pinned individually in |
| 149 | `host_probed_tools.golden.json`, checked whenever the probe succeeds. The |
| 150 | runner also fails if two identical sessions produce different prefixes. |
| 151 | |
| 152 | ### Hook receipts (`hooks` goldens) |
| 153 | |
| 154 | `hooks` is user configuration in the `[[hooks.hooks]]` shape. `{{capture}} NAME` |
| 155 | in a command runs a helper that saves the hook's stdin and every documented |
| 156 | environment variable (`CODEWHALE_*` / `DEEPSEEK_*`, see `HOOK_ENV_CONTRACT`). |
| 157 | The golden records the outcome — `{"admit": {requires_approval, updated_input, |
| 158 | additional_context}}` or `{"refuse": {kind, detail}}` for `tool_call_before`, |
| 159 | the observer results (including exact stdout, stderr and error) for |
| 160 | `tool_call_after` — and, per hook, whether it ran, |
| 161 | its stdin (the schema-1 document, parsed) and its contract environment. The |
| 162 | hook session id and temp paths are masked. POSIX shell: Unix only. |
| 163 | |
| 164 | ## Running a TypeScript implementation against these fixtures |
| 165 | |
| 166 | A TS implementation proves parity by producing byte-identical normalized |
| 167 | output — same masks, same key order (sorted, except prompt bytes) — for every |
| 168 | case, and by passing the Rust runner once it is wired in: |
| 169 | |
| 170 | - **MCP (Phase 2).** Implement `HostMcpDispatch` and add it to `DISPATCHES` in |
| 171 | `crates/tui/src/conformance/mcp.rs`. The transcript server is a real loopback |
| 172 | URL, so the Node host connects to it with the official SDK exactly as it |
| 173 | would to a user's server. The same `mcp/*.golden.json` must pass unchanged. |
| 174 | - **Hooks (Phase 2).** Run the hook orchestration under test with each case's |
| 175 | `hooks` and tool call; the `{{capture}}` helper is plain `sh`, so the |
| 176 | captured stdin/env must match the golden without changes. |
| 177 | - **Tool adapters, slash commands, MCP catalog (Phases 2–3).** Whatever moves, |
| 178 | `prompt/*` must not change unless the change is intended and re-recorded in |
| 179 | the same PR — a prefix change invalidates every user's KV cache. |
| 180 | - **Provider wire adapters (Phase 4, gated).** Replay each `sse/*.sse` with its |
| 181 | case's chunking/framing and emit the stream-event format above. |
| 182 | - **Turn events.** The turn loop stays in Rust (D10); `events/*` is the |
| 183 | regression net around it while its edges move. A host-side consumer of |
| 184 | `EventMsg` can read these goldens as its contract. |
| 185 | |
| 186 | Fields named `detail` carry human-readable error text and are compared exactly, |
| 187 | including by a second implementation. Only the explicitly documented masks and |
| 188 | projections apply. No whitespace, line-ending, error-text, or event-order normalization may |
| 189 | be added to make a drift pass. `.gitattributes` pins fixture checkout bytes to LF. |
| 190 | |
| 191 | ## Coverage boundaries |
| 192 | |
| 193 | The initial corpus contains 28 scenarios: 6 turns, 6 hooks, 2 MCP transcripts, |
| 194 | 3 prefixes, and 11 SSE recordings. Each SSE case must reach its loopback server |
| 195 | and produce a normalized stream event; an incorrectly bound client that fails |
| 196 | before sending is missing evidence. Client construction resolves the case model |
| 197 | through the production route authority. Harness stalls are tested separately |
| 198 | using a hung Engine provider, an unanswered real MCP call, and an open SSE socket. |
| 199 | |
| 200 | This is the Rust reference corpus, not TypeScript parity evidence. No TypeScript |
| 201 | dispatch is registered yet. The MCP cases use the production direct-call seam; |
| 202 | full-turn MCP admission, credentials/OAuth, stdio servers, reconnect/retry races, |
| 203 | and hosted server binaries need separate fixtures. Hook cases cover before-tool |
| 204 | admission and synchronous after-tool observers; message-submit/session hooks, |
| 205 | background observers, cancellation and process-tree teardown are not covered. |
| 206 | The turns do not qualify persistence/resume, nested multi-tool cardinality, |
| 207 | non-draining client backpressure, or every client projection. Slash-command, |
| 208 | script-tool, and migrated adapter calls need their own fixtures before flipping. |
| 209 | The Plan prefix records its advertised catalog and frozen prompt bytes; it |
| 210 | does not prove read-only admission. At this baseline it still advertises |
| 211 | `bash`, `edit` and `write`; full-turn Plan refusal needs separate evidence. |
| 212 | |
| 213 | The prefix runner names unavailable host-probed tools explicitly. A skipped |
| 214 | definition (`code_execution`, `image_ocr`, `js_execution`, `pandoc_convert`) is |
| 215 | unchecked, even when the family passes; a host offering it needs a reviewed |
| 216 | `host_probed_tools.golden.json` entry. Windows turn/hook/prefix fixtures, Linux |
| 217 | replay, hosted CI, native app behavior, and real-provider acceptance are separate |
| 218 | evidence. This corpus supplies none of those receipts by itself. |
| 219 |