返回 last30days-skill
discovery-topic-queue-design-conventions.md
根目录 / docs / solutions / architecture-patterns / discovery-topic-queue-design-conventions.md
1 ---
2 title: "Persistent discovery topic queue: five interlocking design conventions"
3 date: 2026-07-20
4 category: architecture-patterns
5 module: discovery-topic-queue
6 problem_type: architecture_pattern
7 component: database
8 severity: high
9 applies_when:
10 - "Adding default-on local persistence (SQLite, JSON state) hooked onto the end of an expensive pipeline run"
11 - "Building a fuzzy identity layer over LLM-named entities whose names drift across runs"
12 - "Reading a feature toggle in an engine where .env-file values only reach code through env.get_config's keys allowlist"
13 - "Recording per-item state in a loop where a later item could fuzzy-match a row written earlier in the same run"
14 - "Persisting user-set status (covered, dismissed, read) that must survive entity renames"
15 tags:
16 - "discovery-topic-queue"
17 - "fuzzy-matching"
18 - "sqlite-persistence"
19 - "env-allowlist-opt-out"
20 - "two-phase-write"
21 - "covered-status-inheritance"
22 - "guarded-write-hook"
23 - "scoped-db"
24 - "llm-naming-drift"
25 related_components:
26 - "skills/last30days/scripts/store.py"
27 - "skills/last30days/scripts/last30days.py"
28 - "skills/last30days/scripts/lib/env.py"
29 - "tests/test_store.py"
30 - "tests/test_discover_mode.py"
31 ---
32
33
34 # Persistent discovery topic queue: five interlocking design conventions
35
36 ## Context
37
38 PR #852 shipped a persistent topic queue for `/last30days discover`: every real
39 discovery run records which topics it surfaced into a `discovery_topics` table in
40 research.db, so the podcast/X-article pipeline remembers what it has already seen
41 ("surfaced 3rd time") and what the user already produced content for ("marked
42 covered"). This is the design record for that queue - five conventions that were
43 each load-bearing in review, two of them caught as real bugs (one P0). The
44 seed-source corroboration change that landed in the same PR is documented
45 separately in
46 `docs/solutions/design-patterns/ranked-output-confidence-floor-honest-empty-state.md`
47 (section 2b); this doc does not cover it.
48
49 ## Guidance
50
51 ### 1. Default-on, disabled only via the config allowlist - never bare os.environ
52
53 The queue records every real (non-mock) run by default; the literal value `off`
54 disables it. The knob is registered in `env.get_config`'s keys allowlist
55 (`skills/last30days/scripts/lib/env.py:482`):
56
57 ```python
58 # Discovery topic queue (podcast/X-article pipeline memory). Default
59 # ON; the literal value "off" disables queue writes and annotations.
60 ('LAST30DAYS_DISCOVERY_QUEUE', None),
61 ```
62
63 and read from the resolved config dict, never `os.environ`
64 (`skills/last30days/scripts/last30days.py:1312-1314`):
65
66 ```python
67 queue_setting = str(config.get("LAST30DAYS_DISCOVERY_QUEUE") or "").strip().lower()
68 if queue_setting == "off" or not report.topics:
69 return report
70 ```
71
72 WHY: `.env`-file users' values only reach the engine through the `get_config`
73 allowlist merge - a bare `os.environ` read silently ignores them, a documented
74 invisible-failure class in this repo. Scoped runs (`--save-dir`) write the scoped
75 research.db via `store.scoped_db(_scoped_store_db(args))`
76 (`last30days.py:432-437`, `store.py:41-53`), never the global one; `--mock` runs
77 stay 100% side-effect-free (`last30days.py:1505`).
78
79 ### 2. Annotate-only fuzzy matching - a match stamps context, it never merges rows
80
81 `store.match_discovery_topic` tries exact normalized-name match first, then the
82 best entity-overlap candidate - the better of full `entity_key` token overlap and
83 anchor-token overlap - at a conservative floor
84 (`skills/last30days/scripts/store.py:810`, `898-938`):
85
86 ```python
87 DISCOVERY_QUEUE_OVERLAP_THRESHOLD = 0.6
88 ...
89 if best is not None and best_overlap >= DISCOVERY_QUEUE_OVERLAP_THRESHOLD:
90 return dict(best)
91 ```
92
93 A fuzzy match only annotates the rendered card - the `Pipeline: surfaced Nth
94 time, marked covered` line (`skills/last30days/scripts/lib/render.py:153-168`) -
95 and never merges or rewrites queue rows (`store.py:806-809`, `906-907`).
96
97 WHY: with annotate-only semantics a false-positive match costs one noisy line on
98 one card; a false merge would silently collapse two distinct stories into one
99 row and hide one of them forever. The threshold is tunable precisely because
100 mislabeling is recoverable and data loss is not.
101
102 ### 3. Two-phase hook: match ALL topics before recording ANY
103
104 `_annotate_and_record_discovery_queue` computes priors for every topic first,
105 then records surfacings, inside one `store.scoped_db` block
106 (`skills/last30days/scripts/last30days.py:1323-1345`):
107
108 ```python
109 with store.scoped_db(_scoped_store_db(args)):
110 store.init_db()
111 # Phase 1: match EVERY topic before recording ANY. Interleaving
112 # match+record in one loop lets topic N fuzzy-match a same-anchor
113 # sibling row this very run recorded seconds earlier, falsely
114 # annotating a first-ever topic as "surfaced 2nd time".
115 priors = [store.match_discovery_topic(topic.name) for topic in report.topics]
116 # Phase 2: record this run's surfacings. ...
117 for topic, prior in zip(report.topics, priors):
118 ```
119
120 WHY: one report often contains same-anchor siblings ("Gemma 4 chat templates" /
121 "Gemma 4 tool calling fixes"). Interleaved match+record lets topic N fuzzy-match
122 the row topic N-1 wrote seconds earlier, falsely annotating a first-ever topic
123 as a repeat. Caught in review; regression-tested.
124
125 ### 4. Covered inheritance: fresh rows born covered, existing rows never mutated
126
127 `record_discovery_surfacing(inherit_covered_at=...)` makes a fresh row start in
128 `covered` status when its fuzzy-matched prior is covered; the `ON CONFLICT`
129 update path deliberately never touches `status`/`covered_at`
130 (`skills/last30days/scripts/store.py:842-895`):
131
132 ```python
133 status = "covered" if inherit_covered_at else "surfaced"
134 ...
135 ON CONFLICT(normalized_name) DO UPDATE SET
136 surface_count = surface_count + 1,
137 last_surfaced = excluded.last_surfaced,
138 last_run_ref = excluded.last_run_ref,
139 domain = CASE WHEN excluded.domain <> '' THEN excluded.domain ELSE domain END
140 ```
141
142 The caller passes it when a topic's prior is covered
143 (`last30days.py:1334-1344`). Locked by the flip-flop regression test
144 `test_covered_status_survives_judge_rename_across_runs`
145 (`tests/test_store.py:1082-1101`) and by
146 `tests/test_store.py:1060-1079` (ON CONFLICT ignores `inherit_covered_at`).
147
148 WHY: the LLM judge renames the same story across runs; without inheritance a
149 rename forks a fresh uncovered row and the user's covered mark silently
150 evaporates. Without the never-mutate rule, a stale inherit could flip a row the
151 user just changed.
152
153 ### 5. Guarded, synchronous end-of-run write - never crash a finished pipeline
154
155 The hook call in `_run_discover` is wrapped so a broken queue db degrades to a
156 stderr warning and an unannotated report
157 (`skills/last30days/scripts/last30days.py:1505-1515`):
158
159 ```python
160 if not args.mock:
161 try:
162 report = _annotate_and_record_discovery_queue(report, args, config)
163 except (sqlite3.Error, OSError) as exc:
164 # A broken queue db (locked, read-only dir, corrupt) must never
165 # destroy a finished multi-minute pipeline run: warn and render
166 # the report without queue annotations (fields keep defaults).
167 sys.stderr.write(
168 f"[last30days] Warning: discovery queue unavailable ({exc}); "
169 "continuing without queue annotations.\n"
170 )
171 ```
172
173 WHY: unguarded, a locked/read-only/corrupt research.db raises AFTER the
174 multi-minute research pipeline finished and discards all of its output - the PR
175 #852 code review's P0, empirically reproduced. The write also runs synchronously
176 after the pipeline returns (`last30days.py:1308-1310` docstring): it touches
177 disk, so the abandon-on-timeout daemon-thread pattern is forbidden here (see
178 `docs/solutions/logic-errors/non-daemon-executor-threads-defeat-wall-clock-budget.md`).
179
180 ## Why This Matters
181
182 Ranked by blast radius when a convention is violated:
183
184 - Unguarded end-of-run write (5): the whole run's output is destroyed by a
185 bookkeeping failure, and only in degraded environments (locked db, read-only
186 dir), so it ships green and detonates on exactly the machines you cannot see.
187 This was the review's P0.
188 - Interleaved match+record (3): the queue's core promise ("first time you've
189 seen this") is wrong on day one - a first-ever topic gets annotated "surfaced
190 2nd time" by its same-run sibling, and no cross-run test catches it because
191 the corruption happens inside a single run.
192 - Bare os.environ read (1): `.env`-file users cannot turn the queue off; the
193 toggle works in the maintainer's shell and fails invisibly for everyone
194 configuring via file.
195 - Merging on fuzzy match (2): a 0.6-overlap false positive stops being one
196 noisy line and becomes a hidden story - unrecoverable data loss from a
197 heuristic.
198 - Mutating rows or skipping inheritance (4): user covered marks flip-flop with
199 judge naming drift, so the queue re-pitches stories the user already produced,
200 which is the exact failure the queue exists to prevent.
201
202 ## When to Apply
203
204 - Any default-on local persistence bolted onto the end of an expensive pipeline:
205 the write must be guarded (degrade to a warning) and synchronous if it touches
206 disk.
207 - Any fuzzy identity layer over LLM-named entities: keep matching annotate-only,
208 batch all matches before any writes in a run, and inherit user-set status onto
209 fresh rows instead of mutating existing ones.
210 - Any new engine toggle in this repo: register it in `env.get_config`'s keys
211 allowlist and read it from the config dict, never bare `os.environ`.
212
213 ## Examples
214
215 Covered flip-flop, the archetype 3-run scenario (mirrors
216 `tests/test_store.py:1082-1101`):
217
218 1. Run 1 surfaces "Gemma 4 chat templates"; the user records an episode and
219 runs `queue cover "Gemma 4 chat templates"` (row status: covered).
220 2. Run 2's judge names the same story "Gemma 4 template fixes". Exact match
221 misses; fuzzy match (anchor overlap `gemma`/`4` at >= 0.6) finds the covered
222 prior, so the new row is recorded born covered and the card renders
223 `Pipeline: surfaced 2nd time, marked covered` instead of pitching it fresh.
224 3. Run 3 resurfaces "Gemma 4 template fixes"; it exact-matches its own covered
225 row (`covered_at` still the run-1 date). Without convention 4, run 2 would
226 have forked an uncovered row and run 3 would re-pitch a story the user
227 already covered.
228
229 Queue failure behavior: with research.db locked by another process, a discovery
230 run still prints the full rendered report; stderr shows
231 `[last30days] Warning: discovery queue unavailable (database is locked);
232 continuing without queue annotations.` and the cards simply lack Pipeline lines.
233
234 ## Related
235
236 - PR #852 - judged topic names, junk gate, angles, topic queue (this design).
237 - `docs/solutions/design-patterns/ranked-output-confidence-floor-honest-empty-state.md`
238 section 2b - the seed-source corroboration rule from the same PR (not covered
239 here).
240 - `docs/solutions/logic-errors/non-daemon-executor-threads-defeat-wall-clock-budget.md`
241 - why abandon-on-timeout daemon threads are forbidden for disk writers.
242
242 lines MARKDOWN