| 1 | # Catalog refresh |
| 2 | |
| 3 | How CodeWhale keeps model metadata current — what already auto-updates, what |
| 4 | is hand-maintained, and what a scheduled catalog job should (and should not) do. |
| 5 | |
| 6 | Related docs: [`MODEL_PROVIDER_AUDIT.md`](./MODEL_PROVIDER_AUDIT.md), |
| 7 | [`PROVIDERS.md`](./PROVIDERS.md), RFC |
| 8 | [`rfcs/UNIFIED_PROVIDER_LOGIN.md`](./rfcs/UNIFIED_PROVIDER_LOGIN.md). |
| 9 | |
| 10 | --- |
| 11 | |
| 12 | ## Short answer |
| 13 | |
| 14 | | Question | Answer | |
| 15 | |---|---| |
| 16 | | Do users need a special model just to refresh models? | **No.** | |
| 17 | | Does CodeWhale auto-update the public model catalog? | **Yes, at runtime**, from [Models.dev](https://models.dev/catalog.json), ~24 h TTL. | |
| 18 | | Is the offline bundled seed auto-committed in CI? | **Not yet.** Live cache covers running installs; the in-repo seed is still manual / PR-driven. | |
| 19 | | Should an LLM rewrite catalog JSON? | **No.** Ingest is deterministic public JSON. An LLM can *review* a PR, not own the source of truth. | |
| 20 | |
| 21 | --- |
| 22 | |
| 23 | ## Layers (lowest → highest priority) |
| 24 | |
| 25 | Effective precedence for model facts (context, output caps, reasoning, |
| 26 | pricing-ish metadata). Live wins over bundled when present. |
| 27 | |
| 28 | ``` |
| 29 | (5) Legacy static completion lists (DEFAULT_* consts) |
| 30 | — only if catalog has zero rows for the provider |
| 31 | (4) Static code tables |
| 32 | crates/tui/src/models.rs |
| 33 | (3) Bundled offline seeds (NOT competing truth) |
| 34 | crates/config/assets/models_dev.bundled.json |
| 35 | crates/tui/assets/model_catalog.bundled.json |
| 36 | (2) Live Models.dev catalog (preferred when available) |
| 37 | https://models.dev/catalog.json |
| 38 | → disk cache ~/.codewhale/catalog/models-dev-catalog.json |
| 39 | → 24 h TTL |
| 40 | (1) User / custom overrides (pinned models, custom endpoints) |
| 41 | (0) Special: ChatGPT/Codex OAuth roster |
| 42 | ~/.codex/models_cache.json |
| 43 | — bypasses Models.dev for openai-codex only |
| 44 | ``` |
| 45 | |
| 46 | Key code: |
| 47 | |
| 48 | | Piece | Path | Role | |
| 49 | |---|---|---| |
| 50 | | Live fetch + cache | `crates/tui/src/models_dev_live.rs` | Background refresh, TTL, atomic write, freshness status | |
| 51 | | Schema / parse | `crates/config/src/models_dev.rs` | Network-free Models.dev JSON shape | |
| 52 | | Compile + provenance | `crates/config/src/catalog.rs` | Bundled / Live / UserOverride; id normalization | |
| 53 | | Provider lake merge | `crates/tui/src/provider_lake.rs` | Live-over-bundled by `(provider, wire_model_id)` | |
| 54 | | Offline seed asset | `crates/config/assets/models_dev.bundled.json` | Compact offline fallback only (`_meta.role` says so) | |
| 55 | | Validation script | `scripts/catalog_models_dev.py` | Secret-free fetch/validate dry-run (#4117) | |
| 56 | | Script tests | `scripts/catalog_models_dev_test.py` | Offline shape/scrub checks | |
| 57 | |
| 58 | --- |
| 59 | |
| 60 | ## What already auto-updates (runtime) |
| 61 | |
| 62 | When the TUI/runtime starts (and is not disabled): |
| 63 | |
| 64 | 1. Seed pickers from the **on-disk cache** if present (even if stale). |
| 65 | 2. If the cache is missing or older than **24 hours**, **background-fetch** |
| 66 | Models.dev (15 s timeout, explicit CodeWhale user-agent, **no credentials**). |
| 67 | 3. On success: atomic write to |
| 68 | `~/.codewhale/catalog/models-dev-catalog.json` and publish rows into |
| 69 | ProviderLake as `CatalogSource::Live`. |
| 70 | 4. On failure: keep prior cache or fall back to the **bundled** seed. Model |
| 71 | selection never hard-fails because Models.dev is down. |
| 72 | |
| 73 | ### Manual force refresh |
| 74 | |
| 75 | In the TUI: |
| 76 | |
| 77 | ```text |
| 78 | /model refresh |
| 79 | ``` |
| 80 | |
| 81 | That dispatches `AppAction::RefreshModelsDevCatalog` (async; does not block |
| 82 | the composer). Implementation lives under |
| 83 | `crates/tui/src/commands/groups/core/core.rs` and |
| 84 | `crates/tui/src/models_dev_live.rs`. |
| 85 | |
| 86 | ### Env knobs (tests / dogfood / offline) |
| 87 | |
| 88 | | Variable | Effect | |
| 89 | |---|---| |
| 90 | | `CODEWHALE_MODELS_DEV_URL` | Override base URL or full `*.json` catalog URL | |
| 91 | | `CODEWHALE_MODELS_DEV_PATH` | Load catalog from a local file; skip network | |
| 92 | | `CODEWHALE_DISABLE_MODELS_DEV_FETCH` | Truthy → never hit the network (`1` / `true` / `yes` / `on`) | |
| 93 | |
| 94 | Defaults: |
| 95 | |
| 96 | - Catalog URL: `https://models.dev/catalog.json` |
| 97 | - TTL: `24 * 60 * 60` seconds (`DEFAULT_MODELS_DEV_TTL_SECS`) |
| 98 | - Cache file name: `models-dev-catalog.json` under the CodeWhale `catalog` |
| 99 | state dir |
| 100 | |
| 101 | Freshness values exposed for UI / status chips: `bundled` | `live` | `stale` | |
| 102 | `failed`. |
| 103 | |
| 104 | --- |
| 105 | |
| 106 | ## What does **not** auto-update (repo / release) |
| 107 | |
| 108 | These stay hand-maintained or release-lane work until a scheduled PR lands: |
| 109 | |
| 110 | | Surface | Why it drifts | |
| 111 | |---|---| |
| 112 | | `models_dev.bundled.json` | Offline seed; intentionally smaller than full Models.dev | |
| 113 | | `model_catalog.bundled.json` | Compact TUI seed | |
| 114 | | `provider_defaults.rs` / default model IDs | Product choice, not pure catalog dump | |
| 115 | | Static tables in `models.rs` | Fallback heuristics when catalog misses a row | |
| 116 | | Hand-curated `pricing.rs` rows | Vendor billing quirks; not always in Models.dev | |
| 117 | | New `ProviderKind` / wire dialect | Needs code, not only JSON | |
| 118 | |
| 119 | Runtime live refresh **does not** rewrite those files. Users on a recent |
| 120 | install with network still see new Models.dev rows; fresh clones offline, CI |
| 121 | hermetic runs, and first-boot without cache still depend on the seed. |
| 122 | |
| 123 | --- |
| 124 | |
| 125 | ## Maintainer tooling (no LLM) |
| 126 | |
| 127 | ### Validate / dry-run fetch |
| 128 | |
| 129 | ```bash |
| 130 | # Fetch Models.dev + print counts (never writes disk) |
| 131 | python3 scripts/catalog_models_dev.py refresh |
| 132 | |
| 133 | # Validate the committed offline seed still parses as Models.dev-shaped JSON |
| 134 | python3 scripts/catalog_models_dev.py snapshot --check \ |
| 135 | crates/config/assets/models_dev.bundled.json |
| 136 | |
| 137 | # OpenRouter public /models listing (no API key), dry-run only |
| 138 | python3 scripts/catalog_models_dev.py refresh --provider openrouter \ |
| 139 | --sort newest --limit 100 |
| 140 | ``` |
| 141 | |
| 142 | Design constraints of the script (intentional): |
| 143 | |
| 144 | - Public endpoints only — no `Authorization` headers, no API keys. |
| 145 | - Credential-shaped keys are scrubbed if present in remote JSON. |
| 146 | - **Disk writes are disabled** (`--write` / `--write-cache` fail closed). |
| 147 | Staging a new seed is a separate maintainer step so remote JSON is never |
| 148 | blindly committed by automation without review. |
| 149 | |
| 150 | ### Staging a new offline seed (manual) |
| 151 | |
| 152 | 1. Fetch Models.dev to a local file (curl / browser), or use |
| 153 | `CODEWHALE_MODELS_DEV_PATH` against a saved copy. |
| 154 | 2. Scrub to the allowlisted shape (`models`, `providers`, optional `_meta`). |
| 155 | Prefer the script’s public-document rules as the checklist. |
| 156 | 3. Keep seed **compact** — verified defaults for shipped providers, not a |
| 157 | full dump (see `_meta` on the existing asset). |
| 158 | 4. `python3 scripts/catalog_models_dev.py snapshot --check <path>`. |
| 159 | 5. Diff carefully: default wire IDs should stay aligned with |
| 160 | `DEFAULT_*_MODEL` offline. |
| 161 | 6. Open a normal PR. Do not force-push catalog history. |
| 162 | |
| 163 | Optional: use a cheap model **on the PR** to summarize “new / removed / |
| 164 | default-risk” — never as the author of the JSON. |
| 165 | |
| 166 | --- |
| 167 | |
| 168 | ## Recommended scheduled job (not shipped yet) |
| 169 | |
| 170 | Goal: keep the **in-repo offline seed** from rotting, without giving CI write |
| 171 | power over secrets or unsupervised LLM rewrites. |
| 172 | |
| 173 | ```text |
| 174 | cron (daily or weekly) |
| 175 | → fetch Models.dev (public, no keys) |
| 176 | → validate shape + scrub |
| 177 | → compare against crates/config/assets/models_dev.bundled.json |
| 178 | (and optionally report new ids vs provider defaults) |
| 179 | → if material change: open PR |
| 180 | title: chore(catalog): refresh Models.dev offline seed |
| 181 | → optional: agent comments a human-readable diff summary on the PR |
| 182 | ``` |
| 183 | |
| 184 | ### In scope for automation |
| 185 | |
| 186 | - Deterministic catalog ingest from Models.dev |
| 187 | - Secret-free PR diffs |
| 188 | - Drift reports (new model ids, missing defaults, pricing presence) |
| 189 | |
| 190 | ### Out of scope for automation |
| 191 | |
| 192 | - Claude Pro/Max / subscription OAuth “model discovery” (not a supported |
| 193 | third-party path; Anthropic expects API keys for third-party tools) |
| 194 | - LLM-authored edits to `models.rs` / `provider.rs` without review |
| 195 | - Force-pushing `main` or silent asset rewrites on the default branch |
| 196 | - Treating Models.dev as the only truth for OAuth-scoped routes (Codex |
| 197 | roster remains special-cased) |
| 198 | |
| 199 | ### Suggested workflow home |
| 200 | |
| 201 | `CodeWhale/.github/workflows/catalog-refresh.yml` (or similar), reusing |
| 202 | `scripts/catalog_models_dev.py` after a deliberate **write-safe** extension |
| 203 | that only runs in CI with a bot token for PR creation — still not on |
| 204 | `workflow_dispatch` without review if writes land in-repo. |
| 205 | |
| 206 | Nightly today (`/.github/workflows/nightly.yml`) builds release artifacts |
| 207 | only; it does **not** refresh catalogs. |
| 208 | |
| 209 | --- |
| 210 | |
| 211 | ## Do we need a “model dedicated to updating models”? |
| 212 | |
| 213 | **No for the core loop.** |
| 214 | |
| 215 | | Job | Right tool | |
| 216 | |---|---| |
| 217 | | Keep known models/windows/prices from Models.dev fresh for users | Runtime live fetch (already shipped) | |
| 218 | | Keep offline seed + release assets current in git | Scheduled CI → PR (to build) | |
| 219 | | Decide whether to bump a product default model | Human (or agent *review* on the PR) | |
| 220 | | Wire a brand-new provider kind / dialect | Human PR + tests | |
| 221 | |
| 222 | An LLM is optional **review** of a catalog PR. It is a poor **source of |
| 223 | truth** for catalog JSON. |
| 224 | |
| 225 | --- |
| 226 | |
| 227 | ## Auth note (Claude / Anthropic) |
| 228 | |
| 229 | Anthropic model **catalog** refresh does not require Claude Pro/Max OAuth. |
| 230 | Models.dev is public. CodeWhale’s Anthropic route remains **API-key-based** |
| 231 | for inference (`ANTHROPIC_API_KEY`). Do not couple catalog automation to |
| 232 | subscription OAuth or Claude Code identity headers. |
| 233 | |
| 234 | --- |
| 235 | |
| 236 | ## Quick operator checklist |
| 237 | |
| 238 | - [ ] Running install: confirm network not blocked; optional |
| 239 | `/model refresh` after a big vendor launch. |
| 240 | - [ ] Offline / CI hermetic: set `CODEWHALE_DISABLE_MODELS_DEV_FETCH=1` or |
| 241 | point `CODEWHALE_MODELS_DEV_PATH` at a fixture. |
| 242 | - [ ] Before release: `snapshot --check` on the bundled seed; skim |
| 243 | `MODEL_PROVIDER_AUDIT.md` for known drift. |
| 244 | - [ ] After Models.dev adds a major family you ship by default: consider |
| 245 | seed PR + default-model decision separately. |
| 246 | - [ ] Never paste API keys into catalog assets or the automation script env |
| 247 | for Models.dev refresh. |
| 248 | |
| 249 | --- |
| 250 | |
| 251 | ## Issue / design anchors |
| 252 | |
| 253 | - Live Models.dev layer: #4187 |
| 254 | - Bundled seed demoted (not competing truth): #4188 |
| 255 | - Catalog automation script (validate / dry-run): #4117 |
| 256 | - Deeper metadata inventory and drift list: `docs/MODEL_PROVIDER_AUDIT.md` |
| 257 |