| 1 | # Catalog refresh |
| 2 | |
| 3 | > 阅读简体中文版：[zh_hans/CATALOG_REFRESH.md](zh_hans/CATALOG_REFRESH.md)。 |
| 4 | |
| 5 | How Codewhale keeps model metadata current — what already auto-updates, what |
| 6 | is hand-maintained, and what a scheduled catalog job should (and should not) do. |
| 7 | |
| 8 | Related docs: [`PROVIDERS.md`](./PROVIDERS.md), RFC |
| 9 | [`rfcs/UNIFIED_PROVIDER_LOGIN.md`](./rfcs/UNIFIED_PROVIDER_LOGIN.md). |
| 10 | |
| 11 | --- |
| 12 | |
| 13 | ## Short answer |
| 14 | |
| 15 | | Question | Answer | |
| 16 | |---|---| |
| 17 | | Do users need a special model just to refresh models? | **No.** | |
| 18 | | Does Codewhale auto-update the public model catalog? | **Yes, at runtime**, from [Models.dev](https://models.dev/catalog.json), ~24 h TTL. | |
| 19 | | Is the offline bundled seed auto-committed in CI? | **No, but it is generated.** A maintainer runs `seed lock` and `seed render` and opens a PR; CI fails a hand edit (`seed render --check`). | |
| 20 | | Should an LLM rewrite catalog JSON? | **No.** Ingest is deterministic public JSON. An LLM can *review* a PR, not own the source of truth. | |
| 21 | |
| 22 | --- |
| 23 | |
| 24 | ## Layers (lowest → highest priority) |
| 25 | |
| 26 | The shared catalog compiler applies these layers from lowest to highest: |
| 27 | |
| 28 | ``` |
| 29 | 0 bundled Models.dev |
| 30 | 10 live Models.dev |
| 31 | 12 Codewhale corrections (applied to layers 0 and 10 as they load) |
| 32 | 15 verified cloud facts (optional, off by default) |
| 33 | 20 exact provider-owned live roster |
| 34 | 25 Codewhale account roster |
| 35 | 30 config.toml |
| 36 | 40 user overrides |
| 37 | policy DENY (final) |
| 38 | ``` |
| 39 | |
| 40 | Codewhale corrections live in `crates/config/assets/catalog_corrections.json`. |
| 41 | They are field patches in the cloud-facts `ModelFact` shape, applied by the same |
| 42 | patch code to every Models.dev row, offline seed and live refresh alike, so a |
| 43 | correction holds on every install. Use one when an upstream fact is true but |
| 44 | misleading for a Codewhale route: `pricing_withheld` (a reason) clears the price |
| 45 | so the route reports it as unknown, for tiered rates, plan quota and billing |
| 46 | surfaces the catalog cannot tell apart; `max_output` and the other fields patch |
| 47 | limits. Every entry carries its reason. Corrections only fix rows that exist, |
| 48 | never add or hide one, and signed cloud facts can still override them. A |
| 49 | corrected row keeps its own source; a price a correction owns reports |
| 50 | `CatalogSource::CodewhaleBundled` as its price source. Do not hand-edit the offline seed |
| 51 | to hold a value back: a live refresh replaces the seed row, so the hold would |
| 52 | work only offline. |
| 53 | |
| 54 | Cloud facts use the existing compiler and provider lake, as described in |
| 55 | [`CLOUD_FACTS.md`](./CLOUD_FACTS.md). Capability provenance and price provenance |
| 56 | are separate: a capability patch cannot relabel inherited prices. Cloud price |
| 57 | patches replace the entire price block; unspecified token classes stay unknown. |
| 58 | |
| 59 | Route resolution also binds provider kind, configured identity and endpoint. |
| 60 | A fresh provider-owned roster is authoritative for its exact scope. Explicit |
| 61 | model selections remain explicit. Codex account observations/native cache and |
| 62 | Ollama endpoint tags keep their dedicated availability rules; a public catalog |
| 63 | row does not prove that an account can call that model. The installed Codex |
| 64 | `account/read` and `model/list` path is documented in |
| 65 | [`PROVIDERS.md`](./PROVIDERS.md). |
| 66 | |
| 67 | Legacy completion lists remain a last fallback where no applicable catalog |
| 68 | exists. Bundled seeds and static transport/billing rules remain release-owned; |
| 69 | refreshing catalog metadata does not introduce a new wire dialect or change |
| 70 | credential/billing ownership. |
| 71 | |
| 72 | Key code: |
| 73 | |
| 74 | | Piece | Path | Role | |
| 75 | |---|---|---| |
| 76 | | Live fetch + cache | `crates/tui/src/models_dev_live.rs` | Background refresh, TTL, atomic write, freshness status | |
| 77 | | Schema / parse | `crates/config/src/models_dev.rs` | Network-free Models.dev JSON shape | |
| 78 | | Compile + provenance | `crates/config/src/catalog.rs` | Ordered sources, independent price provenance, policy deny, id normalization | |
| 79 | | Provider lake merge | `crates/tui/src/provider_lake.rs` | Shared catalog projection with exact route-scoped provider authority | |
| 80 | | Offline seed asset | `crates/config/assets/models_dev.bundled.json` | Compact offline fallback only (`_meta.role` says so) | |
| 81 | | Codewhale corrections | `crates/config/assets/catalog_corrections.json` | Field patches applied to every Models.dev row (`crates/config/src/catalog/corrections.rs`) | |
| 82 | | Validation script | `scripts/catalog_models_dev.py` | Secret-free fetch/validate dry-run (#4117) | |
| 83 | | Script tests | `scripts/catalog_models_dev_test.py` | Offline shape/scrub checks | |
| 84 | |
| 85 | --- |
| 86 | |
| 87 | ## What already auto-updates (runtime) |
| 88 | |
| 89 | When the TUI/runtime starts (and is not disabled): |
| 90 | |
| 91 | 1. Seed pickers from the **on-disk cache** if present (even if stale). |
| 92 | 2. If the cache is missing or older than **24 hours**, **background-fetch** |
| 93 | Models.dev (15 s timeout, explicit Codewhale user-agent, **no credentials**). |
| 94 | 3. On success: atomic write to |
| 95 | `~/.codewhale/catalog/models-dev-catalog.json` and publish rows into |
| 96 | ProviderLake as `CatalogSource::ModelsDevLive` — layer 10, carrying no |
| 97 | endpoint fingerprint. Models.dev is a public catalog describing a model, so |
| 98 | a refreshed row is treated exactly like the layer-0 seed it supersedes and |
| 99 | stays correctable by layer 15. `CatalogSource::Live` is reserved for a |
| 100 | provider's own credential-scoped `/models` answer at layer 20. |
| 101 | 4. On failure: keep prior cache or fall back to the **bundled** seed. Model |
| 102 | selection never hard-fails because Models.dev is down. |
| 103 | |
| 104 | ### Manual force refresh |
| 105 | |
| 106 | In the TUI: |
| 107 | |
| 108 | ```text |
| 109 | /model refresh |
| 110 | ``` |
| 111 | |
| 112 | That dispatches `AppAction::RefreshModelsDevCatalog` (async; does not block |
| 113 | the composer). When admitted cloud-facts settings are enabled, it also requests |
| 114 | a cloud refresh; hard-disable and trust-key checks still apply. Implementation lives under |
| 115 | `crates/tui/src/commands/groups/core/core.rs` and |
| 116 | `crates/tui/src/models_dev_live.rs`. |
| 117 | |
| 118 | ### Env knobs (tests / dogfood / offline) |
| 119 | |
| 120 | | Variable | Effect | |
| 121 | |---|---| |
| 122 | | `CODEWHALE_MODELS_DEV_URL` | Override base URL or full `*.json` catalog URL | |
| 123 | | `CODEWHALE_MODELS_DEV_PATH` | Load catalog from a local file; skip network | |
| 124 | | `CODEWHALE_DISABLE_MODELS_DEV_FETCH` | Truthy → never hit the network (`1` / `true` / `yes` / `on`) | |
| 125 | |
| 126 | Defaults: |
| 127 | |
| 128 | - Catalog URL: `https://models.dev/catalog.json` |
| 129 | - TTL: `24 * 60 * 60` seconds (`DEFAULT_MODELS_DEV_TTL_SECS`) |
| 130 | - Cache file name: `models-dev-catalog.json` under the Codewhale `catalog` |
| 131 | state dir |
| 132 | |
| 133 | Freshness values exposed for UI / status chips: `bundled` | `live` | `stale` | |
| 134 | `failed`. |
| 135 | |
| 136 | --- |
| 137 | |
| 138 | ## What does **not** auto-update (repo / release) |
| 139 | |
| 140 | These stay hand-maintained or release-lane work until a scheduled PR lands: |
| 141 | |
| 142 | | Surface | Why it drifts | |
| 143 | |---|---| |
| 144 | | `models_dev.bundled.json` | Offline seed, generated from a reviewed spec and a pinned lock (see below); refreshed by PR, not at runtime | |
| 145 | | `provider_descriptors.json` default model IDs | Product choice, not pure catalog dump; constant projections are generated | |
| 146 | | `catalog_corrections.json` `reviewed` | Intrinsic/selector/transport compatibility facts with exact source receipts; generated into the same seed | |
| 147 | | Rust pricing policy | Vendor billing windows, CNY conversion, withheld/tiered behavior; pure reference observations live in the reviewed supplement | |
| 148 | | New `ProviderKind` / wire dialect | Needs code, not only JSON | |
| 149 | |
| 150 | Runtime live refresh **does not** rewrite those files. Users on a recent |
| 151 | install with network still see new Models.dev rows; fresh clones offline, CI |
| 152 | hermetic runs, and first-boot without cache still depend on the seed. |
| 153 | |
| 154 | --- |
| 155 | |
| 156 | ## Maintainer tooling (no LLM) |
| 157 | |
| 158 | ### Validate / dry-run fetch |
| 159 | |
| 160 | ```bash |
| 161 | # Fetch Models.dev + print counts (never writes disk) |
| 162 | python3 scripts/catalog_models_dev.py refresh |
| 163 | |
| 164 | # Validate the committed offline seed still parses as Models.dev-shaped JSON |
| 165 | python3 scripts/catalog_models_dev.py snapshot --check \ |
| 166 | crates/config/assets/models_dev.bundled.json |
| 167 | |
| 168 | # OpenRouter public /models listing (no API key), dry-run only |
| 169 | python3 scripts/catalog_models_dev.py refresh --provider openrouter \ |
| 170 | --sort newest --limit 100 |
| 171 | ``` |
| 172 | |
| 173 | Design constraints of the script (intentional): |
| 174 | |
| 175 | - Public endpoints only — no `Authorization` headers, no API keys. |
| 176 | - Credential-shaped keys are scrubbed if present in remote JSON. |
| 177 | - `refresh` and `snapshot` never write (`--write` / `--write-cache` fail |
| 178 | closed). The one write path is `seed lock`, which pins only the rows the |
| 179 | spec references, projected onto allowlisted fields. |
| 180 | |
| 181 | ### Regenerating the offline seed (#6396) |
| 182 | |
| 183 | `crates/config/assets/models_dev.bundled.json` is generated. Never edit it by |
| 184 | hand: CI runs `seed render --check` and fails on any difference. |
| 185 | |
| 186 | | File | Holds | Edited by | |
| 187 | |---|---|---| |
| 188 | | `scripts/catalog/models_dev_seed.toml` | Which upstream rows to carry, their Codewhale provider id, wire id, default, canonical join, and the few curated rows upstream does not list | Hand, reviewed | |
| 189 | | `scripts/catalog/models_dev_seed.lock.json` | The referenced upstream rows, allowlisted, plus the source URL, fetch time and sha256 | `seed lock` only | |
| 190 | | `crates/config/assets/catalog_corrections.json` | Deliberate holds: withheld prices, clamped limits, reasoning controls | Hand, reviewed; applies online too | |
| 191 | | `crates/config/assets/catalog_corrections.json` `reviewed` | Source-preserved intrinsic facts, scoped aliases, completion references, public labels/source-support dates, pure route facts and reference prices | Hand, reviewed; source receipts retained | |
| 192 | | `crates/config/assets/models_dev.bundled.json` | The rendered seed including the reviewed supplement | `seed render` only | |
| 193 | |
| 194 | The spec selects and maps; it cannot state a value that disagrees with |
| 195 | upstream (unknown keys are refused). If an upstream value is wrong for a |
| 196 | Codewhale route, add a correction instead. Corrections apply to both the seed |
| 197 | and live rows; a hold made only by hand-editing the seed would vanish on the |
| 198 | first live refresh. |
| 199 | |
| 200 | 1. `python3 scripts/catalog_models_dev.py seed lock --dry-run` prints the |
| 201 | review report: field changes per row, corrections that upstream now |
| 202 | agrees with (delete them), and upstream models not carried. It fails when |
| 203 | a referenced row disappeared upstream, or a curated row now exists |
| 204 | upstream (switch it to a derived row). |
| 205 | 2. Edit the spec or the corrections as the report requires. |
| 206 | 3. `python3 scripts/catalog_models_dev.py seed lock` writes the lock. |
| 207 | 4. `python3 scripts/catalog_models_dev.py seed render` writes the seed. |
| 208 | 5. Check that default wire IDs still match `DEFAULT_*_MODEL`, run the |
| 209 | catalog tests, and open a PR with the report in its body. |
| 210 | |
| 211 | Optional: use a cheap model to summarize “new / removed / default-risk” in |
| 212 | the PR body — never as the author of the JSON. |
| 213 | |
| 214 | --- |
| 215 | |
| 216 | ## Recommended scheduled job (not shipped yet) |
| 217 | |
| 218 | Goal: keep the **in-repo offline seed** from rotting, without giving CI write |
| 219 | power over secrets or unsupervised LLM rewrites. |
| 220 | |
| 221 | ```text |
| 222 | cron (daily or weekly) |
| 223 | → fetch Models.dev (public, no keys) |
| 224 | → validate shape + scrub |
| 225 | → compare against crates/config/assets/models_dev.bundled.json |
| 226 | (and optionally report new ids vs provider defaults) |
| 227 | → if material change: open PR |
| 228 | title: chore(catalog): refresh Models.dev offline seed |
| 229 | → optional: include an agent-written, human-readable diff summary in the PR body |
| 230 | ``` |
| 231 | |
| 232 | Such a job would run `seed lock` and `seed render` and open the PR. A PR |
| 233 | opened with the default `GITHUB_TOKEN` does not trigger CI, so it needs a bot |
| 234 | token or GitHub App, which a maintainer has to provision. |
| 235 | |
| 236 | ### In scope for automation |
| 237 | |
| 238 | - Deterministic catalog ingest from Models.dev |
| 239 | - Secret-free PR diffs |
| 240 | - Drift reports (new model ids, missing defaults, pricing presence) |
| 241 | |
| 242 | ### Out of scope for automation |
| 243 | |
| 244 | - Claude Pro/Max / subscription OAuth “model discovery” (not a supported |
| 245 | third-party path; Anthropic expects API keys for third-party tools) |
| 246 | - LLM-authored edits to `models.rs` / `provider.rs` without review |
| 247 | - Force-pushing `main` or silent asset rewrites on the default branch |
| 248 | - Treating Models.dev as the only truth for OAuth-scoped routes (Codex |
| 249 | roster remains special-cased) |
| 250 | |
| 251 | ### Suggested workflow home |
| 252 | |
| 253 | `CodeWhale/.github/workflows/catalog-refresh.yml` (or similar), reusing |
| 254 | `scripts/catalog_models_dev.py` after a deliberate **write-safe** extension |
| 255 | that only runs in CI with a bot token for PR creation — still not on |
| 256 | `workflow_dispatch` without review if writes land in-repo. |
| 257 | |
| 258 | Nightly today (`/.github/workflows/nightly.yml`) builds release artifacts |
| 259 | only; it does **not** refresh catalogs. |
| 260 | |
| 261 | --- |
| 262 | |
| 263 | ## Do we need a “model dedicated to updating models”? |
| 264 | |
| 265 | **No for the core loop.** |
| 266 | |
| 267 | | Job | Right tool | |
| 268 | |---|---| |
| 269 | | Keep known models/windows/prices from Models.dev fresh for users | Runtime live fetch (already shipped) | |
| 270 | | Keep offline seed + release assets current in git | Scheduled CI → PR (to build) | |
| 271 | | Decide whether to bump a product default model | Human (or agent *review* on the PR) | |
| 272 | | Wire a brand-new provider kind / dialect | Human PR + tests | |
| 273 | |
| 274 | An LLM is optional **review** of a catalog PR. It is a poor **source of |
| 275 | truth** for catalog JSON. |
| 276 | |
| 277 | --- |
| 278 | |
| 279 | ## Auth note (Claude / Anthropic) |
| 280 | |
| 281 | Anthropic model **catalog** refresh does not require Claude Pro/Max OAuth. |
| 282 | Models.dev is public. Codewhale’s Anthropic route remains **API-key-based** |
| 283 | for inference (`ANTHROPIC_API_KEY`). Do not couple catalog automation to |
| 284 | subscription OAuth or Claude Code identity headers. |
| 285 | |
| 286 | --- |
| 287 | |
| 288 | ## Quick operator checklist |
| 289 | |
| 290 | - [ ] Running install: confirm network not blocked; optional |
| 291 | `/model refresh` after a big vendor launch. |
| 292 | - [ ] Offline / CI hermetic: set `CODEWHALE_DISABLE_MODELS_DEV_FETCH=1` or |
| 293 | point `CODEWHALE_MODELS_DEV_PATH` at a fixture. |
| 294 | - [ ] Before release: `seed lock --dry-run` to see how far the offline seed |
| 295 | has drifted from Models.dev; re-lock by PR if it matters. Skim |
| 296 | `PROVIDERS.md` for known drift. |
| 297 | - [ ] After Models.dev adds a major family you ship by default: consider |
| 298 | seed PR + default-model decision separately. |
| 299 | - [ ] Never paste API keys into catalog assets or the automation script env |
| 300 | for Models.dev refresh. |
| 301 | |
| 302 | --- |
| 303 | |
| 304 | ## Issue / design anchors |
| 305 | |
| 306 | - Live Models.dev layer: #4187 |
| 307 | - Bundled seed demoted (not competing truth): #4188 |
| 308 | - Catalog automation script (validate / dry-run): #4117 |
| 309 | - Generated offline seed and runtime corrections: #6396 |
| 310 | - Deeper metadata inventory and drift list: the `codewhale-ops` repo |
| 311 | |
| 312 | ### Retired unscoped metadata reader |
| 313 | |
| 314 | The former models crate cache reader and its separate bundled asset, and the |
| 315 | TUI-only model registry, are retired. Existing installed legacy cache files are |
| 316 | preserved and are never imported as provider or public-label authority. |
| 317 | `config::catalog` owns the immutable compiled intrinsic projection; Engine's |
| 318 | existing provider lake and scoped catalog cache still own live/account/config |
| 319 | facts. Identical wire names at different endpoints do not create a canonical |
| 320 | join, a public label, or an unscoped price. The bundled freshness clock uses the |
| 321 | actual seed lock fetch timestamp and rechecks the current clock on each query. |
| 322 | |
| 323 | Compatibility completion lists reference the provider descriptor defaults and |
| 324 | catalog groups rather than repeating their values. Kimi's generation default |
| 325 | remains distinct from direct/membership route limits. Unknown capabilities and |
| 326 | name-suffix budgets remain explicitly unverified. Website model dates describe |
| 327 | source support, with the prior proven dates retained in the same reviewed owner; |
| 328 | they do not assert a provider release date or current API availability. |
| 329 |