| 1 | # Model Lab Roadmap |
| 2 | |
| 3 | Model Lab is the planned open-model workbench for Codewhale. The north star is |
| 4 | simple: Codewhale should make open-source and open-weight models practical in |
| 5 | terminal coding workflows across every provider that offers them. Model Lab is how |
| 6 | those models become discoverable, evaluable, routable, servable, and exportable |
| 7 | without weakening the current terminal-agent contract: local workspace control, |
| 8 | explicit provider auth, approval gates, and clear privacy boundaries. |
| 9 | |
| 10 | This document is roadmap language. Some worksets below are roadmap-only. |
| 11 | |
| 12 | ## Implemented Today |
| 13 | |
| 14 | - DeepSeek is the first-class default provider today, with `deepseek-v4-pro`, |
| 15 | `deepseek-v4-flash`, streaming thinking blocks, Fin routing, `DEEPSEEK_*` |
| 16 | environment variables, and `~/.deepseek` config compatibility. |
| 17 | - OpenRouter, Novita, Fireworks, NVIDIA NIM, AtlasCloud, Wanjie Ark, Hugging |
| 18 | Face Inference Providers, generic OpenAI-compatible endpoints, SGLang, vLLM, |
| 19 | and Ollama are supported provider paths where their IDs appear in |
| 20 | `/provider`, `codewhale --provider`, or `codewhale models`. |
| 21 | - Hugging Face Inference Providers are available through the |
| 22 | OpenAI-compatible router at `https://router.huggingface.co/v1`. Select the |
| 23 | route with `huggingface`, `hugging-face`, `hugging_face`, or `hf`; configure |
| 24 | `HUGGINGFACE_API_KEY` or `HF_TOKEN` for auth. |
| 25 | - Model auto-routing chooses a concrete DeepSeek model and thinking level per |
| 26 | turn. It is not a TUI mode. |
| 27 | - Fin is the fast `deepseek-v4-flash` thinking-off path for routing, |
| 28 | summaries, cheap checks, RLM child calls, wakeup verification, and |
| 29 | binary-completion checks. |
| 30 | - Self-hosted OpenAI-compatible endpoints can be used through SGLang, vLLM, |
| 31 | Ollama, or the generic `openai` provider configuration. |
| 32 | |
| 33 | ## Still Planned |
| 34 | |
| 35 | - A native Hugging Face Hub browser, model passport picker, or direct Hub search |
| 36 | workflow. The OpenAI-compatible Hugging Face Inference Providers route is |
| 37 | implemented separately as a chat provider. |
| 38 | - Built-in Hugging Face model card, dataset, adapter, safetensors, Spaces, or |
| 39 | Jobs workflows. |
| 40 | - Native Unsloth, NeMo, or Arcee integrations. |
| 41 | - A dedicated Model Lab UI tab. |
| 42 | - Built-in eval leaderboards, hosted observability, or training-infrastructure |
| 43 | orchestration. |
| 44 | |
| 45 | Until those land, use the provider paths above, MCP servers, or external |
| 46 | workflows explicitly configured by the user. |
| 47 | |
| 48 | ## Model Lab Principle |
| 49 | |
| 50 | Model Lab should help users answer practical questions: |
| 51 | |
| 52 | - Which model should handle this turn? |
| 53 | - Which open or open-weight model can I run locally or through a trusted |
| 54 | provider? |
| 55 | - Which provider offers this model with the latency, price, context window, |
| 56 | license, and privacy posture I need? |
| 57 | - What did this model cost, how did it perform, and what data left my machine? |
| 58 | - Can I reproduce, export, or self-host the route? |
| 59 | |
| 60 | It should never hide provider boundaries, silently upload local artifacts, or |
| 61 | describe a model as available before Codewhale can actually route to it. |
| 62 | |
| 63 | ## Hugging Face Workset |
| 64 | |
| 65 | Implemented today: |
| 66 | |
| 67 | - Hugging Face Inference Providers as an explicit OpenAI-compatible router |
| 68 | provider, selected with `huggingface`, `hugging-face`, `hugging_face`, or |
| 69 | `hf`. |
| 70 | - Model IDs are sent to the router exactly as selected, including |
| 71 | org-prefixed Hugging Face model IDs. |
| 72 | |
| 73 | Planned scope: |
| 74 | |
| 75 | - Hub API auth and model discovery. |
| 76 | - Model cards, licenses, tags, safetensors metadata, adapters, and dataset |
| 77 | links surfaced in a terminal-friendly way. |
| 78 | - Native Hub browser and model-passport metadata on top of the already separate |
| 79 | Hugging Face Inference Providers chat route. |
| 80 | - Hugging Face Jobs as an optional remote execution path for user-approved |
| 81 | experiments. |
| 82 | |
| 83 | Non-goal for now: claiming native Hub search, model passports, Spaces/Jobs, or |
| 84 | Model Lab UI exists before those surfaces are implemented in code. |
| 85 | The inference-provider API key does not imply Hub browsing/export, upload, or |
| 86 | Jobs authorization. |
| 87 | |
| 88 | ## Unsloth Workset |
| 89 | |
| 90 | Planned scope: |
| 91 | |
| 92 | - Fine-tuning recipes and adapter workflows for users who already own the data |
| 93 | and compute path. |
| 94 | - Export guidance that keeps dataset, adapter, and checkpoint locations explicit. |
| 95 | - Compatibility notes for models that can return to local serving or a hosted |
| 96 | OpenAI-compatible endpoint. |
| 97 | |
| 98 | ## NeMo Workset |
| 99 | |
| 100 | Planned scope: |
| 101 | |
| 102 | - Training and alignment workflow notes for users operating NVIDIA-centric |
| 103 | infrastructure. |
| 104 | - Clear boundaries between NVIDIA NIM inference support that exists today and |
| 105 | future NeMo training or customization workflows. |
| 106 | |
| 107 | ## Arcee Workset |
| 108 | |
| 109 | Planned scope: |
| 110 | |
| 111 | - Small-model routing and specialization experiments. |
| 112 | - Exportable routes that make it clear when a task is handled by a smaller |
| 113 | model, Fin, or full DeepSeek reasoning. |
| 114 | |
| 115 | ## Serving Workset |
| 116 | |
| 117 | Planned scope: |
| 118 | |
| 119 | - Better local and private serving ergonomics for SGLang, vLLM, Ollama, and |
| 120 | OpenAI-compatible gateways. |
| 121 | - Health checks, model listing, context-window metadata, and route validation. |
| 122 | - No silent network exposure: public endpoints must be configured explicitly. |
| 123 | |
| 124 | ## Eval Workset |
| 125 | |
| 126 | Implemented authoring foundation: |
| 127 | |
| 128 | - Provider-neutral `WorkflowSearchSpec` validation and deterministic freeze |
| 129 | receipts for an experimental-search option within Workflow. The freeze binds |
| 130 | the baseline, requested and resolved model names, public evidence, and |
| 131 | evaluator identity before candidate admission. |
| 132 | - The best-of-N Workflow starter can generate 2–16 structured, independent |
| 133 | worktree candidates with cache-stable shared instructions and a read-only |
| 134 | review. This is generation/review evidence, not runtime-owned hard-gate proof. |
| 135 | |
| 136 | Planned scope: |
| 137 | |
| 138 | - Reproducible task suites for coding, review, docs, release checks, and |
| 139 | long-context workflows. |
| 140 | - Side-by-side route comparisons where the exact model, provider, thinking |
| 141 | level, prompt, and tool policy are captured. |
| 142 | - Runtime-owned hard gates and command scoring after worker write authority is |
| 143 | revoked; clean-baseline replay; duplicate-patch detection; multi-round |
| 144 | Pareto/diversity promotion; aggregate receipts over Fleet receipts; and a |
| 145 | Workflow-panel leaderboard. No winner is applied or merged automatically. |
| 146 | |
| 147 | ## Observability Workset |
| 148 | |
| 149 | Planned scope: |
| 150 | |
| 151 | - Local-first traces for turn routing, tool calls, approvals, cost, cache |
| 152 | behavior, and context pressure. |
| 153 | - Export rules that redact secrets and require explicit user action before data |
| 154 | leaves the machine. |
| 155 | |
| 156 | ## Training Infra Workset |
| 157 | |
| 158 | Planned scope: |
| 159 | |
| 160 | - Recipes for dataset preparation, adapter training, artifact naming, and |
| 161 | promotion into serving. |
| 162 | - Separation between local/private artifacts and anything published to a hub or |
| 163 | registry. |
| 164 | |
| 165 | ## Privacy And Export Rules |
| 166 | |
| 167 | - Local files, prompts, transcripts, traces, model outputs, eval results, |
| 168 | adapters, datasets, and checkpoints should remain local unless the user |
| 169 | explicitly chooses a provider or export destination. |
| 170 | - Provider auth must remain explicit. `DEEPSEEK_*`, OpenRouter, |
| 171 | `HUGGINGFACE_API_KEY` / `HF_TOKEN`, and self-hosted credentials should not be |
| 172 | inferred from unrelated config. |
| 173 | - Exportable artifacts should include provenance: source model, provider, |
| 174 | route, tool policy, eval inputs, and redaction status. |
| 175 | - Public sharing, hosted telemetry, sponsorship badges, and external branding |
| 176 | require maintainer approval. |
| 177 |