| 1 | # `telemetry-ingest` — Codewhale's first-party telemetry endpoint |
| 2 | |
| 3 | A Cloudflare Worker that accepts the batches described in |
| 4 | [`docs/TELEMETRY.md`](../docs/TELEMETRY.md) and writes them to Workers Analytics |
| 5 | Engine. One POST route. No response body on any path. No client IP, anywhere, |
| 6 | ever. |
| 7 | |
| 8 | It lives here and not in `web/` because the site is a separate deploy with its |
| 9 | own build (Next.js via OpenNext); this is a single 13 KiB script with no assets, |
| 10 | and coupling the two would mean a telemetry change rebuilding the marketing site. |
| 11 | |
| 12 | **Deployed and live** at `https://telemetry.codewhale.net/v1/telemetry`, which |
| 13 | is the shipped default for `telemetry_endpoint`. workers.dev is disabled; that |
| 14 | hostname is the only way in. |
| 15 | |
| 16 | That default decides where an *already enabled* session's batches go. It is not |
| 17 | a consent change: Codewhale telemetry is still opt-in and off by default, and |
| 18 | nothing is collected until the first-run notice is answered with Enable. A user |
| 19 | who wants to stay enabled and contact nobody sets `telemetry_endpoint = ""`, |
| 20 | which writes batches to `$CODEWHALE_HOME/telemetry/dryrun.jsonl` instead. |
| 21 | |
| 22 | --- |
| 23 | |
| 24 | ## The one property that matters |
| 25 | |
| 26 | `docs/TELEMETRY.md` publishes: |
| 27 | |
| 28 | > Batches are **IP-stripped at ingest**. No IP is stored, logged, or joined to |
| 29 | > `install_id`. |
| 30 | |
| 31 | This Worker is the whole of what makes that sentence true. There is no other |
| 32 | component. So: |
| 33 | |
| 34 | - `src/index.ts` reads exactly two request headers — `content-type` and |
| 35 | `content-length` — and nothing else, ever. |
| 36 | - It never touches the `cf` property of the request, so country, colo, city, |
| 37 | region, ASN and coordinates are never in scope. |
| 38 | - `src/datapoint.ts` builds every stored row, and it cannot see the request at |
| 39 | all: its input type is the validated batch body. |
| 40 | - Nothing logs. `invocation_logs` is off in `wrangler.jsonc`, because Cloudflare |
| 41 | describes those as "enriched with information available to Cloudflare in the |
| 42 | context of the invocation" — exactly the class of automatic per-request record |
| 43 | this service promises not to keep. |
| 44 | - `test/no-ip.test.ts` reads the shipped source as text and fails the build if |
| 45 | any of those names appears, if the set of headers read grows past two, if a |
| 46 | `console.*` call is added, or if a `Response` is ever constructed with a body. |
| 47 | That file is the only place in this directory where the forbidden header names |
| 48 | are written down. A later edit cannot add one quietly. |
| 49 | |
| 50 | Debugging without an IP is a solved problem: the schema carries `os`, `arch`, |
| 51 | `libc`, `surface`, `app_version` and `git_sha`, which is what crash triage |
| 52 | actually needs. |
| 53 | |
| 54 | ## What it stores |
| 55 | |
| 56 | Everything in [`docs/TELEMETRY.md`](../docs/TELEMETRY.md) and nothing else. The |
| 57 | validator in `src/schema.ts` is a **closed** field set: an unexpected key |
| 58 | anywhere in the batch — envelope, event, `counters`, `errors`, `turn_wall` — |
| 59 | rejects the whole batch with `400`. That is the point of the design. A future |
| 60 | client bug that starts attaching a path, a prompt, or a customer's provider |
| 61 | table name gets refused by the server rather than quietly stored. |
| 62 | |
| 63 | `test/schema-doc.test.ts` parses the field names and enum spellings back out of |
| 64 | `docs/TELEMETRY.md` and asserts set equality against the validator, and |
| 65 | `test/ingest.test.ts` posts `crates/telemetry/tests/golden/v1.json` — the |
| 66 | client's own pinned v1 wire form — and asserts it is accepted byte for byte and |
| 67 | that deleting *any* key from it is rejected. The doc, the Rust client, and this |
| 68 | endpoint cannot drift apart without a red test. |
| 69 | |
| 70 | ### Column layout |
| 71 | |
| 72 | One Analytics Engine data point per event. A batch carries at most 200 events |
| 73 | (`BATCH_MAX_EVENTS`) and Analytics Engine allows 250 data points per invocation, |
| 74 | so a conforming batch never needs a second pass. |
| 75 | |
| 76 | The layout is **positional and append-only** — Analytics Engine columns are |
| 77 | `blob1..blob20` / `double1..double20`, and the names below exist only in the SQL |
| 78 | you write. Renumbering silently rewrites every historical query. To add a field, |
| 79 | take the next free slot. |
| 80 | |
| 81 | | column | contents | |
| 82 | |---|---| |
| 83 | | `index1` | `install_id` — random v4 UUID, client-rotated every 90 days. The only identifier in the schema. | |
| 84 | | `blob1` | `event` — `install_or_upgrade` \| `session_start` \| `session_end` \| `panic` | |
| 85 | | `blob2` | `surface` | |
| 86 | | `blob3` | `os` | |
| 87 | | `blob4` | `arch` | |
| 88 | | `blob5` | `libc` | |
| 89 | | `blob6` | `app_version` | |
| 90 | | `blob7` | `git_sha`, `''` for every locally built binary | |
| 91 | | `blob8` | `tty` — `'true'` \| `'false'` | |
| 92 | | `blob9` | `install_kind` (`install_or_upgrade` only) | |
| 93 | | `blob10` | `previous_version` (`install_or_upgrade` only) | |
| 94 | | `blob11` | `session_source` (`session_start` only) | |
| 95 | | `blob12` | `duration_bucket` (`session_end` only) | |
| 96 | | `blob13` | `exit_class` (`session_end` only) | |
| 97 | | `blob14` | `cold_start_bucket` (`session_end` only; `''` on surfaces that do not measure it) | |
| 98 | | `blob15` | `providers`, comma-joined, already sorted and deduplicated | |
| 99 | | `blob16` | `panic_site` (`panic` only) — a `crates/…` path or the literal `<dep>` | |
| 100 | | `blob17` | `sent_at` — the *batch* timestamp. Events carry none. | |
| 101 | | `double1..10` | `counters`: `turns`, `tool_calls`, `fleet_dispatch`, `workflow_run`, `subagent_spawn`, `mcp_server_connected`, `memory_search`, `approval_modal_shown`, `approval_auto_allowed`, `command_palette_open` | |
| 102 | | `double11..16` | `errors`: `auth_preflight_failed`, `provider_http_4xx`, `provider_http_5xx`, `tool_denied_by_policy`, `tool_timeout`, `network_error` | |
| 103 | | `double17..20` | `turn_wall`: `lt_5s`, `5_30s`, `30_120s`, `gte_120s` | |
| 104 | |
| 105 | Columns not relevant to an event are `''` / `0`. `tty` is a blob because the 20 |
| 106 | doubles are exactly used by the three numeric structs — Analytics Engine's |
| 107 | ceiling is 20. |
| 108 | |
| 109 | ### What it structurally cannot store |
| 110 | |
| 111 | Not "does not"; **cannot**, given the code as written: |
| 112 | |
| 113 | - **The client IP, and anything derived from it** — never read. See above. |
| 114 | - **Any geo** — country, colo, city, region, ASN, coordinates, timezone. |
| 115 | - **Any key the schema does not name.** Unknown key ⇒ `400` for the entire |
| 116 | batch, so there is no path from an unexpected field to storage. |
| 117 | - **Any free-form string.** The published schema has no free-form string type and |
| 118 | no open-keyed map. Every field is an integer, a boolean, or a closed enum, |
| 119 | except `app_version`, `git_sha` and `panic_site` — each of which has a regex |
| 120 | here, so a path, a prompt, a URL, or a branch name fails the shape check. |
| 121 | - **A provider table name.** `providers` entries must be lowercase hyphenated ids |
| 122 | and the array must be sorted and deduplicated; `acme_internal_gateway` is |
| 123 | rejected. (See "known gap" below.) |
| 124 | - **A panic message.** Only `panic_site`, and only inside the `crates/` |
| 125 | allowlist or the literal `<dep>`. |
| 126 | - **Per-event timestamps.** There are none in the schema; only `sent_at`, |
| 127 | per batch. |
| 128 | - **Response content.** Every response is a bare status with a `null` body, so |
| 129 | the endpoint cannot echo back what it received or what it holds. |
| 130 | |
| 131 | **Known gap, stated plainly.** `providers` is the one field whose *value* space |
| 132 | this endpoint cannot close. The authoritative list is |
| 133 | `codewhale_config::provider::all_providers()`, a Rust registry with no generated |
| 134 | artifact to read, and hard-coding a copy here would drift into silently dropping |
| 135 | a real user's route. The client closes it (`Event::is_bounded` → |
| 136 | `is_known_provider_id`) before the POST is made; the server enforces the shape a |
| 137 | closed `&'static str` enum can produce, plus the doc's sorted-and-deduplicated |
| 138 | rule. If a generated provider-id list ever lands in the repo, wire it in here. |
| 139 | |
| 140 | ### Retention |
| 141 | |
| 142 | Cloudflare stores Analytics Engine data for **three months**, and that is not |
| 143 | configurable — a ceiling rather than a policy, since no setting could make it |
| 144 | longer. `docs/TELEMETRY.md` states it. |
| 145 | |
| 146 | --- |
| 147 | |
| 148 | ## Deploy |
| 149 | |
| 150 | Live. The commands below are the ones that produced the current deployment and |
| 151 | the ones that will produce the next one. |
| 152 | |
| 153 | ```sh |
| 154 | cd telemetry-ingest |
| 155 | npm install |
| 156 | npm test # 87 tests, including the doc weld and the IP guard |
| 157 | npx wrangler deploy --dry-run --outdir=.wrangler/dry-run # no account touched |
| 158 | npx wrangler deploy # <- the only command that publishes anything |
| 159 | ``` |
| 160 | |
| 161 | The `routes` block in `wrangler.jsonc` binds it to `telemetry.codewhale.net` as |
| 162 | a custom domain, so the endpoint URL is |
| 163 | `https://telemetry.codewhale.net/v1/telemetry`. The workers.dev subdomain is |
| 164 | disabled: that hostname is the only way in. |
| 165 | |
| 166 | Verified against the live endpoint before the client default was changed: the |
| 167 | client's golden batch returns `204` with a zero-byte body; an unknown key |
| 168 | returns `400`; `GET` returns `405`; a wrong content type returns `415`; a `POST` |
| 169 | to `/` returns `404`. Reading back from Analytics Engine returned exactly two |
| 170 | rows — `session_start` and `session_end`, carrying `install_id`, `surface=tui`, |
| 171 | `os=macos` — the documented shape and nothing else. |
| 172 | |
| 173 | **Re-run the verification below after any deploy.** The client default now |
| 174 | points here, so a regression in this Worker is a regression in a promise |
| 175 | `docs/TELEMETRY.md` makes to users. |
| 176 | |
| 177 | ### Analytics Engine dataset setup |
| 178 | |
| 179 | There is none. The dataset named in `wrangler.jsonc` |
| 180 | (`codewhale_telemetry`) is created implicitly on the first successful |
| 181 | `writeDataPoint`, so there is nothing to provision ahead of the deploy. Confirm |
| 182 | it exists after the first batch: |
| 183 | |
| 184 | ```sh |
| 185 | curl "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/analytics_engine/sql" \ |
| 186 | --header "Authorization: Bearer $CF_API_TOKEN" \ |
| 187 | --data "SHOW TABLES" |
| 188 | ``` |
| 189 | |
| 190 | The token needs **Account → Account Analytics → Read**. Querying is out of band |
| 191 | through this API; the Worker itself has no read path at all. |
| 192 | |
| 193 | ### Rate limiting |
| 194 | |
| 195 | `ratelimits` binding, 20 POSTs per 60s, **keyed on `install_id`** from the |
| 196 | validated batch body. Never on a network address — an IP-keyed limiter would |
| 197 | mean this Worker handles IPs, which is the whole thing it must not do. That is a |
| 198 | weaker limiter (an `install_id.json` can be rewritten between POSTs) and it is |
| 199 | the right trade: Cloudflare's edge already absorbs volumetric abuse, and the |
| 200 | client only flushes twice per session anyway (a startup drain at most once every |
| 201 | six hours, and one three-second attempt at shutdown). |
| 202 | |
| 203 | ### Size cap |
| 204 | |
| 205 | `MAX_BODY_BYTES` is 72 KiB, computed rather than guessed. A conforming client |
| 206 | sends at most 200 events totalling at most 65536 bytes (`BATCH_MAX_EVENTS` and |
| 207 | `BATCH_MAX_BYTES` in `crates/telemetry/src/actor.rs`, both hard ceilings because |
| 208 | `parse_events` breaks *before* crossing them), plus 199 commas and ~375 bytes of |
| 209 | envelope keys and values — 66110 bytes worst case. 72 KiB is ~11% headroom. |
| 210 | |
| 211 | The 512-record / 256 KiB rings in `crates/telemetry/src/buffer.rs` are the *disk* |
| 212 | cap, not the wire cap: a full ring drains as three batches, never as one POST. |
| 213 | |
| 214 | `content-length` is checked first as a cheap reject, but it is client-supplied, |
| 215 | so the real bound is enforced while reading the body and the stream is cancelled |
| 216 | the moment it goes over. |
| 217 | |
| 218 | --- |
| 219 | |
| 220 | ## The two queries |
| 221 | |
| 222 | Both are one query each, which is what the column layout was chosen for. Run |
| 223 | them against the SQL API: |
| 224 | |
| 225 | ```sh |
| 226 | query() { |
| 227 | curl -sS "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/analytics_engine/sql" \ |
| 228 | --header "Authorization: Bearer $CF_API_TOKEN" --data "$1" |
| 229 | } |
| 230 | ``` |
| 231 | |
| 232 | `_sample_interval` says how many original rows a stored row represents; |
| 233 | Analytics Engine downsamples high-volume indexes, so every count is weighted by |
| 234 | it rather than using bare `count()`. |
| 235 | |
| 236 | ### (a) How many installs and sessions |
| 237 | |
| 238 | ```sql |
| 239 | SELECT |
| 240 | count(DISTINCT index1) AS installs, |
| 241 | sumIf(_sample_interval, blob1 = 'session_start') AS sessions_started, |
| 242 | sumIf(_sample_interval, blob1 = 'session_end') AS sessions_ended, |
| 243 | sumIf(_sample_interval, blob9 = 'install') AS first_installs, |
| 244 | sumIf(_sample_interval, blob9 = 'upgrade') AS upgrades, |
| 245 | sumIf(_sample_interval, blob13 = 'clean') AS clean_exits |
| 246 | FROM codewhale_telemetry |
| 247 | WHERE timestamp > NOW() - INTERVAL '7' DAY |
| 248 | ``` |
| 249 | |
| 250 | Read `installs` as what the published doc says it is and nothing more: the id |
| 251 | rotates every 90 days and is regenerated whenever the telemetry directory is |
| 252 | cleared, so **no count derived from `install_id` is a user count**. It is a lower |
| 253 | bound on distinct machine-installs seen in the window, and it undercounts a |
| 254 | returning user across a rotation. |
| 255 | |
| 256 | Add `, blob6` to `SELECT` and `GROUP BY blob6` to cut by `app_version`; `blob3` |
| 257 | for OS, `blob2` for surface. |
| 258 | |
| 259 | ### (b) Which error classes and panic sites dominate |
| 260 | |
| 261 | ```sql |
| 262 | SELECT |
| 263 | blob16 AS panic_site, |
| 264 | sum(_sample_interval) AS rows, |
| 265 | sum(double11 * _sample_interval) AS auth_preflight_failed, |
| 266 | sum(double12 * _sample_interval) AS provider_http_4xx, |
| 267 | sum(double13 * _sample_interval) AS provider_http_5xx, |
| 268 | sum(double14 * _sample_interval) AS tool_denied_by_policy, |
| 269 | sum(double15 * _sample_interval) AS tool_timeout, |
| 270 | sum(double16 * _sample_interval) AS network_error |
| 271 | FROM codewhale_telemetry |
| 272 | WHERE timestamp > NOW() - INTERVAL '7' DAY |
| 273 | AND blob1 IN ('session_end', 'panic') |
| 274 | GROUP BY panic_site |
| 275 | ORDER BY rows DESC |
| 276 | ``` |
| 277 | |
| 278 | One query, two answers, because `panic_site` is `''` on every non-panic row: |
| 279 | |
| 280 | - the **`panic_site = ''` row** carries the six error-class totals across all |
| 281 | `session_end` events in the window — that is the error ranking; |
| 282 | - every **other row** is one panic site, ranked by how often it fired. |
| 283 | |
| 284 | Sessions that ended in a panic are visible either way: `blob13 = 'panic'` on the |
| 285 | `session_end` row, and the `panic` event carries the site. |
| 286 | |
| 287 | --- |
| 288 | |
| 289 | ## Verifying no IP is stored |
| 290 | |
| 291 | Three checks, in increasing order of how convincing they are. |
| 292 | |
| 293 | **1. The source cannot ask for it.** `npm test` runs `test/no-ip.test.ts`, which |
| 294 | greps the shipped source. Prove the guard is live by adding a |
| 295 | `request.headers.get("CF-Connecting-IP")` line to `src/index.ts` and re-running — |
| 296 | three tests go red — then revert. |
| 297 | |
| 298 | **2. Every stored column is accounted for.** The schema is closed and the layout |
| 299 | above is exhaustive; there is no free slot an address could occupy. Confirm the |
| 300 | deployed dataset has exactly the columns you expect: |
| 301 | |
| 302 | ```sh |
| 303 | query "SELECT * FROM codewhale_telemetry LIMIT 1 FORMAT JSON" |
| 304 | ``` |
| 305 | |
| 306 | The result carries `dataset`, `timestamp`, `_sample_interval`, `index1`, and the |
| 307 | `blob*`/`double*` columns. There is no address column, because Analytics Engine |
| 308 | has no implicit one — every column is written by `writeDataPoint`. |
| 309 | |
| 310 | **3. Search the stored data for an address shape.** After the first real |
| 311 | batches, this returns zero rows: |
| 312 | |
| 313 | ```sql |
| 314 | SELECT count() AS suspicious |
| 315 | FROM codewhale_telemetry |
| 316 | WHERE timestamp > NOW() - INTERVAL '7' DAY |
| 317 | AND ( |
| 318 | match(index1, '\\d+\\.\\d+\\.\\d+\\.\\d+') |
| 319 | OR match(concat(blob1, blob2, blob3, blob4, blob5, blob6, blob7, blob8, |
| 320 | blob9, blob10, blob11, blob12, blob13, blob14, blob15, |
| 321 | blob16, blob17), '\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|:[0-9a-f]{1,4}:') |
| 322 | ) |
| 323 | ``` |
| 324 | |
| 325 | Also confirm nothing is being logged: with `invocation_logs` off and no |
| 326 | `console.*` call in the source, **Workers Logs → this Worker** should show no |
| 327 | per-request entries at all. |
| 328 | |
| 329 | --- |
| 330 | |
| 331 | ## Local development |
| 332 | |
| 333 | ```sh |
| 334 | npm install |
| 335 | npm test # vitest, 87 tests |
| 336 | npm run typecheck # tsc --noEmit |
| 337 | npm run check # wrangler deploy --dry-run — touches no account |
| 338 | npm run dev # wrangler dev --local |
| 339 | ``` |
| 340 | |
| 341 | `wrangler dev --local` runs the real `workerd` with local Analytics Engine and |
| 342 | rate-limit bindings. Post the client's own golden batch at it: |
| 343 | |
| 344 | ```sh |
| 345 | curl -i -X POST http://127.0.0.1:8787/v1/telemetry \ |
| 346 | -H 'content-type: application/json' \ |
| 347 | --data-binary @../crates/telemetry/tests/golden/v1.json |
| 348 | # HTTP/1.1 204 No Content, zero-byte body |
| 349 | |
| 350 | curl -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8787/v1/telemetry |
| 351 | # 405, Allow: POST |
| 352 | ``` |
| 353 | |
| 354 | ### Responses |
| 355 | |
| 356 | | status | when | |
| 357 | |---|---| |
| 358 | | `204` | accepted, zero-byte body, no headers | |
| 359 | | `400` | not JSON, or fails the published schema — including any unknown key | |
| 360 | | `404` | POST to a path other than `/v1/telemetry` | |
| 361 | | `405` | any method other than POST, on any path (`Allow: POST`) | |
| 362 | | `413` | body over `MAX_BODY_BYTES` | |
| 363 | | `415` | content type is not `application/json` | |
| 364 | | `429` | this `install_id` is over the rate limit | |
| 365 | | `500` | internal error; nothing was written | |
| 366 | |
| 367 | Every one of them has an empty body. The client |
| 368 | (`crates/telemetry/src/client.rs`) reads only the status class and drops the |
| 369 | batch on anything that is not 2xx — no retry, no backoff, no re-queue — so a |
| 370 | rejection is invisible to the user by construction, and a 5xx here can never |
| 371 | become a client-visible error. That is what lets this endpoint fail closed: |
| 372 | when in doubt, refuse the batch. |
| 373 |