diff --git a/.gsd/DECISIONS.md b/.gsd/DECISIONS.md index 09bdc67d8..3f398cb71 100644 --- a/.gsd/DECISIONS.md +++ b/.gsd/DECISIONS.md @@ -50,3 +50,6 @@ | D042 | M003/S04 | pattern | shouldUseWorktreeIsolation override parameter | Accept optional overridePrefs for testability | loadEffectiveGSDPreferences computes PROJECT_PREFERENCES_PATH at module load time from process.cwd(). chdir-based test fixtures cannot influence it. Override parameter enables reliable testing. | Yes — if preference loading becomes dynamic | | D043 | M003/S04 | pattern | validatePreferences exported | Export from preferences.ts for direct test access | Was module-private. Tests need to call it directly without full file-loading pipeline. No downstream consumers affected. | No | | D044 | M003/S05 | pattern | Self-heal strategy for merge failures | Detect real conflicts immediately (skip retry), retry only transient failures once | Real conflicts will fail identically on retry — wasting time. Transient failures (stale index, leftover merge state) recover after abort+reset. Fast escalation for conflicts, automatic recovery for everything else. | Yes — if retry proves useful for some conflict types | +| D045 | M004 | arch | SQLite provider strategy | Tiered chain: node:sqlite → better-sqlite3 → null | node:sqlite available on Node 22.5+ (our target), better-sqlite3 as fallback for older Node, null for graceful degradation. DbAdapter normalizes API differences. | Yes — if node:sqlite stabilizes and better-sqlite3 path can be dropped | +| D046 | M004 | arch | createWorktree sync/async for DB copy | Keep synchronous, use copyFileSync | Memory-db made createWorktree async for dynamic imports, but copyWorktreeDb is purely sync (copyFileSync). Static import + isDbAvailable() guard avoids async cascade through createAutoWorktree and auto.ts call sites. | No | +| D047 | M004 | arch | Port strategy | Adapt to current architecture, not blind merge | 145 commits divergence, auto.ts decomposed into 6 modules. Memory-db code is reference — capabilities ported into current file structure (auto-prompts.ts, auto-dispatch.ts, etc.), not cherry-picked. | No | diff --git a/.gsd/PROJECT.md b/.gsd/PROJECT.md index 8c492d555..934fcb61c 100644 --- a/.gsd/PROJECT.md +++ b/.gsd/PROJECT.md @@ -2,7 +2,7 @@ ## What This Is -A pi coding agent extension (GSD — "Get Stuff Done") that provides structured planning, auto-mode execution, and project management for autonomous coding sessions. Includes proactive secret management, browser automation tools for UI verification, and worktree-isolated git architecture for zero-friction autonomous execution. +A pi coding agent extension (GSD — "Get Stuff Done") that provides structured planning, auto-mode execution, and project management for autonomous coding sessions. Includes proactive secret management, browser automation tools for UI verification, worktree-isolated git architecture for zero-friction autonomous execution, and SQLite-backed surgical context injection for token-efficient prompt assembly. ## Core Value @@ -21,11 +21,13 @@ The GSD extension is fully functional with: - Worktree-isolated git architecture: auto-worktree per milestone, --no-ff slice merges, milestone squash to main, preference-gated isolation modes, self-healing git repair, doctor git health checks, full e2e test coverage - Auto-worktree lifecycle: `auto-worktree.ts` module creates isolated worktrees per milestone (`milestone/` branches), wired into auto.ts startAuto/resume/stop with split-brain prevention - Branch-per-slice git model with squash merge to main (legacy mode, supported via `git.isolation: "branch"` preference) +- Decomposed auto-mode: `auto-prompts.ts` (prompt builders), `auto-dispatch.ts` (unit→prompt routing), `auto-recovery.ts` (timeout/crash recovery), `auto-worktree.ts` (worktree lifecycle) ## Architecture / Key Patterns - **Extension model**: pi extensions register tools, commands, hooks via `ExtensionAPI` - **State machine**: `auto.ts` drives `dispatchNextUnit()` which reads disk state and dispatches fresh sessions +- **Dispatch pipeline**: `auto-dispatch.ts` resolves phase → unit type + prompt via `resolveDispatch()`. Prompt builders live in `auto-prompts.ts`. - **Secrets gate**: `startAuto()` checks `getManifestStatus()` before first dispatch - **Disk-driven state**: `.gsd/` files are the source of truth, `STATE.md` is derived cache - **File parsing**: `files.ts` has markdown parsers for all GSD file types @@ -43,3 +45,4 @@ See `.gsd/REQUIREMENTS.md` for the explicit capability contract, requirement sta - [x] M001: Proactive Secret Management — Front-loaded API key collection into planning so auto-mode runs uninterrupted (10 requirements validated) - [x] M002: Browser Tools Performance & Intelligence — Module decomposition, action pipeline optimization, sharp-based screenshots, form intelligence, intent-ranked retrieval, semantic actions, 108-test suite (12 requirements validated) - [x] M003: Worktree-Isolated Git Architecture — Auto-worktree per milestone, --no-ff slice merges, milestone squash to main, preferences + backwards compat, self-healing git repair, doctor health checks, full e2e test suite (13 requirements validated) +- [ ] M004: SQLite Context Store — Surgical context injection via SQLite-backed query layer, replacing whole-file prompt dumps with scoped DB queries for ≥30% token savings diff --git a/.gsd/REQUIREMENTS.md b/.gsd/REQUIREMENTS.md index 802d6c64e..86fabc74e 100644 --- a/.gsd/REQUIREMENTS.md +++ b/.gsd/REQUIREMENTS.md @@ -4,7 +4,148 @@ This file is the explicit capability and coverage contract for the project. ## Active -(No active requirements — all M003 requirements validated.) +### R045 — SQLite DB layer with tiered provider chain +- Class: core-capability +- Status: active +- Description: A SQLite abstraction layer that tries `node:sqlite` (Node 22.5+), falls back to `better-sqlite3`, then to null. A thin `DbAdapter` interface normalizes API differences. Schema init creates decisions, requirements, artifacts tables plus filtered views. WAL mode on file-backed databases. +- Why it matters: The foundation for surgical context injection. Without a queryable store, prompts must dump entire files. +- Source: execution (memory-db port) +- Primary owning slice: M004/S01 +- Supporting slices: none +- Validation: unmapped +- Notes: Port from memory-db worktree `gsd-db.ts`. Tiered provider chain proven on Node 22.20.0. `node:sqlite` returns null-prototype rows — DbAdapter normalizes via spread. + +### R046 — Graceful degradation when SQLite unavailable +- Class: continuity +- Status: active +- Description: When no SQLite provider loads, all query functions return empty results and all prompt builders fall back to `inlineGsdRootFile` filesystem loading. No crash, no visible error. +- Why it matters: SQLite must be optional. Users on exotic platforms or old Node versions must not be blocked. +- Source: execution (memory-db port) +- Primary owning slice: M004/S01 +- Supporting slices: M004/S03 +- Validation: unmapped +- Notes: Every query function guards with `isDbAvailable()` + try/catch. Every prompt builder falls back to existing `inlineGsdRootFile`. + +### R047 — Auto-migration from markdown to DB on first run +- Class: core-capability +- Status: active +- Description: When auto-mode starts on a project with `.gsd/` markdown files but no `gsd.db`, silently import all artifact types into a fresh DB. Idempotent — safe to re-run. +- Why it matters: Existing projects must transparently gain DB benefits without manual migration. +- Source: execution (memory-db port) +- Primary owning slice: M004/S02 +- Supporting slices: M004/S01 +- Validation: unmapped +- Notes: Port from memory-db `md-importer.ts`. Custom parsers for DECISIONS.md pipe-table format and REQUIREMENTS.md section/bullet format. Hierarchy walker for milestones → slices → tasks. + +### R048 — Round-trip fidelity for all artifact types +- Class: quality-attribute +- Status: active +- Description: Importing markdown into DB and regenerating markdown produces field-identical output. No data loss, no format drift. +- Why it matters: Dual-write means DB→markdown generation must be faithful. Format drift corrupts the human-readable artifacts. +- Source: execution (memory-db port) +- Primary owning slice: M004/S02 +- Supporting slices: M004/S06 +- Validation: unmapped +- Notes: Port from memory-db. Custom parsers and generators must produce/consume identical formats. + +### R049 — Surgical prompt injection via DB queries +- Class: core-capability +- Status: active +- Description: All prompt builders in `auto-prompts.ts` use scoped DB queries instead of whole-file `inlineGsdRootFile` for decisions, requirements, and project context. Decisions filtered by milestone, requirements filtered by slice ownership. +- Why it matters: This is the core value — smaller, more relevant prompts mean better agent reasoning and fewer wasted tokens. +- Source: user +- Primary owning slice: M004/S03 +- Supporting slices: M004/S01, M004/S02 +- Validation: unmapped +- Notes: Port from memory-db DB-aware helpers. Must be rewired into current `auto-prompts.ts` (not the old monolithic auto.ts). 19 `inlineGsdRootFile` calls to replace across 11 prompt builders. + +### R050 — Dual-write keeping markdown and DB in sync +- Class: continuity +- Status: active +- Description: After each dispatch unit completes and auto-commits, re-import modified markdown files into the DB. Structured LLM tools write to DB first, then regenerate markdown. Both directions stay synchronized. +- Why it matters: Markdown files are the human-readable source of truth. The DB is the query index. They must agree. +- Source: execution (memory-db port) +- Primary owning slice: M004/S03 +- Supporting slices: M004/S06 +- Validation: unmapped +- Notes: Re-import in `handleAgentEnd` after auto-commit. DB-first write in structured tools triggers markdown generation. + +### R051 — Token measurement with before/after comparison +- Class: operability +- Status: active +- Description: `promptCharCount` and `baselineCharCount` fields added to `UnitMetrics`. Measurement wired into all `snapshotUnitMetrics` call sites. Baseline = full markdown content. Prompt = DB-scoped content. Difference = token savings. +- Why it matters: Proves the ≥30% savings claim with real data. Enables ongoing monitoring of prompt efficiency. +- Source: execution (memory-db port) +- Primary owning slice: M004/S04 +- Supporting slices: M004/S03 +- Validation: unmapped +- Notes: Port from memory-db. Module-scoped measurement vars reset at top of `dispatchNextUnit`. + +### R052 — DB-first state derivation with filesystem fallback +- Class: core-capability +- Status: active +- Description: `deriveState()` queries the artifacts table for file content when DB is available, replacing the batch file-parse step. File discovery still uses disk. Falls back to filesystem when DB unavailable. +- Why it matters: Faster state derivation on large projects. Consistent with DB-first architecture. +- Source: execution (memory-db port) +- Primary owning slice: M004/S04 +- Supporting slices: M004/S01, M004/S02 +- Validation: unmapped +- Notes: Port from memory-db. File discovery (which milestones/slices/tasks exist) stays on disk. Only content loading switches to DB. + +### R053 — Worktree DB copy on creation +- Class: integration +- Status: active +- Description: When a worktree is created, copy `gsd.db` from the source project into the worktree's `.gsd/` directory. Skip WAL/SHM files. Non-fatal on failure. +- Why it matters: Worktrees need their own DB with the project's current state. Without a copy, the worktree starts with no DB context. +- Source: execution (memory-db port) +- Primary owning slice: M004/S05 +- Supporting slices: M004/S01 +- Validation: unmapped +- Notes: Port from memory-db `copyWorktreeDb`. Keep `createWorktree` synchronous — `copyFileSync` is sufficient. Guard with `isDbAvailable()`. + +### R054 — Worktree DB merge reconciliation +- Class: integration +- Status: active +- Description: When a worktree merges back (slice or milestone), ATTACH the worktree's DB and reconcile rows: INSERT OR REPLACE in a transaction with conflict detection by content column comparison. +- Why it matters: The worktree may have added decisions, requirements, or artifacts that the main DB doesn't have. +- Source: execution (memory-db port) +- Primary owning slice: M004/S05 +- Supporting slices: M004/S01 +- Validation: unmapped +- Notes: Port from memory-db `reconcileWorktreeDb`. ATTACH/DETACH pattern with try/finally for cleanup. + +### R055 — Structured LLM tools for decisions/requirements/summaries +- Class: core-capability +- Status: active +- Description: Three tools registered: `gsd_save_decision` (auto-assigns D-numbers, writes to DB + regenerates DECISIONS.md), `gsd_update_requirement` (verifies existence, updates DB + regenerates REQUIREMENTS.md), `gsd_save_summary` (writes artifact to DB + disk). +- Why it matters: Eliminates the markdown-then-parse roundtrip. LLM writes structured data directly, guaranteeing parseable output. +- Source: execution (memory-db port) +- Primary owning slice: M004/S06 +- Supporting slices: M004/S03 +- Validation: unmapped +- Notes: Port from memory-db. DB-first write pattern: upsert → fetch all → generate markdown → write file. + +### R056 — /gsd inspect command for DB diagnostics +- Class: operability +- Status: active +- Description: A `/gsd inspect` slash command that dumps schema version, table row counts, and recent entries from each table. +- Why it matters: When things go wrong, the user needs visibility into DB state without running raw SQL. +- Source: execution (memory-db port) +- Primary owning slice: M004/S06 +- Supporting slices: M004/S01 +- Validation: unmapped +- Notes: Port from memory-db. Autocomplete for subcommands (decisions, requirements, artifacts, all). + +### R057 — ≥30% token savings on planning/research dispatches +- Class: quality-attribute +- Status: active +- Description: Surgical prompt injection delivers ≥30% fewer prompt characters compared to whole-file loading, measured on mature projects with multiple milestones, decisions, and requirements. +- Why it matters: The primary user-visible value of the entire DB architecture. If savings aren't real, the complexity isn't justified. +- Source: user +- Primary owning slice: M004/S07 +- Supporting slices: M004/S03, M004/S04 +- Validation: unmapped +- Notes: Memory-db proved: 52.2% plan-slice, 66.3% decisions-only, 32.2% research composite, 42.4% lifecycle. Must re-prove against current codebase. ## Validated @@ -516,11 +657,24 @@ This file is the explicit capability and coverage contract for the project. | R042 | core-capability | deferred | none | none | unmapped | | R043 | quality-attribute | deferred | none | none | unmapped | | R044 | anti-feature | out-of-scope | none | none | n/a | +| R045 | core-capability | active | M004/S01 | none | unmapped | +| R046 | continuity | active | M004/S01 | M004/S03 | unmapped | +| R047 | core-capability | active | M004/S02 | M004/S01 | unmapped | +| R048 | quality-attribute | active | M004/S02 | M004/S06 | unmapped | +| R049 | core-capability | active | M004/S03 | M004/S01, M004/S02 | unmapped | +| R050 | continuity | active | M004/S03 | M004/S06 | unmapped | +| R051 | operability | active | M004/S04 | M004/S03 | unmapped | +| R052 | core-capability | active | M004/S04 | M004/S01, M004/S02 | unmapped | +| R053 | integration | active | M004/S05 | M004/S01 | unmapped | +| R054 | integration | active | M004/S05 | M004/S01 | unmapped | +| R055 | core-capability | active | M004/S06 | M004/S03 | unmapped | +| R056 | operability | active | M004/S06 | M004/S01 | unmapped | +| R057 | quality-attribute | active | M004/S07 | M004/S03, M004/S04 | unmapped | ## Coverage Summary -- Active requirements: 0 -- Mapped to slices: 0 +- Active requirements: 13 +- Mapped to slices: 13 - Validated: 35 - Deferred: 5 - Out of scope: 4 diff --git a/.gsd/milestones/M004/M004-CONTEXT.md b/.gsd/milestones/M004/M004-CONTEXT.md new file mode 100644 index 000000000..651908833 --- /dev/null +++ b/.gsd/milestones/M004/M004-CONTEXT.md @@ -0,0 +1,126 @@ +# M004: SQLite Context Store — Surgical Prompt Injection + +**Gathered:** 2026-03-15 +**Status:** Ready for planning + +## Project Description + +Port the completed memory-db worktree's SQLite-backed context store into the current GSD codebase. The memory-db work (7 slices, 21 requirements validated, 293 tests) was built against a pre-v2.12.0 codebase that has since diverged significantly — 145 commits on main including auto.ts decomposition, worktree architecture overhaul, and extensive refactoring. This is a port, not a merge. + +## Why This Milestone + +The current prompt assembly dumps entire files (DECISIONS.md, REQUIREMENTS.md, PROJECT.md) into every dispatch prompt regardless of relevance. On a mature project with 40+ decisions and 30+ requirements, most of that context is irrelevant to the active slice. A SQLite query layer enables surgical injection — only the decisions scoped to this milestone, only the requirements owned by this slice. The user's emphasis: "super fast context ingestion" — the DB is the mechanism for being "very, very surgically" selective about what context each task sees. + +## User-Visible Outcome + +### When this milestone is complete, the user can: + +- Run auto-mode and see ≥30% smaller prompts with only relevant context injected +- Use `gsd_save_decision`, `gsd_update_requirement`, `gsd_save_summary` tool calls that bypass markdown parsing +- Run `/gsd inspect` to see DB state for diagnostics +- Start auto-mode on an existing project and have gsd.db appear silently with all artifacts imported + +### Entry point / environment + +- Entry point: `/gsd auto` CLI command, structured LLM tools during dispatch, `/gsd inspect` slash command +- Environment: local dev (Node 22.5+, runs in pi agent process) +- Live dependencies involved: none (SQLite is embedded, no external services) + +## Completion Class + +- Contract complete means: DB opens, queries return scoped data, prompt builders use DB queries, tests pass +- Integration complete means: full auto-mode cycle runs with DB-backed context injection, dual-write keeps markdown in sync, worktree lifecycle copies/reconciles DB +- Operational complete means: existing projects migrate transparently, graceful fallback when SQLite unavailable, token savings measured and ≥30% + +## Final Integrated Acceptance + +To call this milestone complete, we must prove: + +- A full auto-mode dispatch cycle (research → plan → execute → complete) produces correct prompts with scoped context from the DB +- An existing project with markdown artifacts silently migrates to DB on first run with zero data loss +- Token measurement shows ≥30% savings on planning/research units +- The system works identically (via fallback) when SQLite is unavailable +- TypeScript compiles clean, all existing tests pass, new DB test suite passes + +## Risks and Unknowns + +- `auto-prompts.ts` has 11 prompt builders with 19 `inlineGsdRootFile` calls — rewiring must preserve existing prompt structure and fallback behavior +- `handleAgentEnd` in `auto.ts` has new post-unit-hook machinery since memory-db was built — dual-write re-import must integrate without disrupting hooks/doctor/rebuildState sequence +- `worktree-manager.ts` `createWorktree` is sync on main — DB copy must work synchronously (decision: use `copyFileSync`, keep sync) +- `node:sqlite` is experimental in Node 22 — API could change, but the DbAdapter abstraction insulates against this +- Memory-db's markdown parsers for DECISIONS.md and REQUIREMENTS.md are custom (not using `files.ts`) — must verify they handle current file formats + +## Existing Codebase / Prior Art + +- `src/resources/extensions/gsd/auto-prompts.ts` — 880 lines, 11 `build*Prompt()` functions, 19 `inlineGsdRootFile` calls. This is where surgical injection happens. +- `src/resources/extensions/gsd/auto-dispatch.ts` — `resolveDispatch()` maps units to prompt builders. Imports from `auto-prompts.ts`. +- `src/resources/extensions/gsd/auto.ts` — `startAuto()`, `handleAgentEnd()`, `dispatchNextUnit()`. DB init/migration goes in startup, re-import in handleAgentEnd. +- `src/resources/extensions/gsd/state.ts` — `deriveState()` — 587 lines. DB-first content loading replaces batch file parse. +- `src/resources/extensions/gsd/metrics.ts` — `UnitMetrics` interface, `snapshotUnitMetrics()`. Add `promptCharCount`/`baselineCharCount`. +- `src/resources/extensions/gsd/worktree-manager.ts` — `createWorktree()` (sync), `mergeWorktreeToMain()`. DB copy/reconcile hooks here. +- `src/resources/extensions/gsd/index.ts` — tool registrations. 3 new structured tools. +- `src/resources/extensions/gsd/commands.ts` — slash command registration. `/gsd inspect`. +- `src/resources/extensions/gsd/types.ts` — needs Decision/Requirement interfaces. +- `.gsd/worktrees/memory-db/` — the source worktree with all memory-db implementation. Reference code lives here. + +### Memory-db source modules to port: +- `.gsd/worktrees/memory-db/src/resources/extensions/gsd/gsd-db.ts` — 750 lines, SQLite abstraction layer +- `.gsd/worktrees/memory-db/src/resources/extensions/gsd/context-store.ts` — 195 lines, query layer + formatters +- `.gsd/worktrees/memory-db/src/resources/extensions/gsd/md-importer.ts` — 526 lines, markdown parsers + migration orchestrator +- `.gsd/worktrees/memory-db/src/resources/extensions/gsd/db-writer.ts` — 337 lines, DB→markdown generators + DB-first write helpers +- `.gsd/worktrees/memory-db/src/resources/extensions/gsd/tests/` — 13 test files covering all DB capabilities + +> See `.gsd/DECISIONS.md` for all architectural and pattern decisions — it is an append-only register; read it during planning, append to it during execution. + +## Relevant Requirements + +- R045–R057 — all 13 active requirements map to this milestone's 7 slices + +## Scope + +### In Scope + +- SQLite DB layer with tiered provider chain (node:sqlite → better-sqlite3 → null) +- Auto-migration from markdown files to DB +- Surgical prompt injection via DB queries in all prompt builders +- Dual-write keeping markdown and DB in sync (both directions) +- Token measurement with before/after comparison in UnitMetrics +- DB-first state derivation in deriveState() +- Worktree DB copy on creation and merge reconciliation +- 3 structured LLM tools (gsd_save_decision, gsd_update_requirement, gsd_save_summary) +- /gsd inspect slash command +- Full test suite for all DB capabilities + +### Out of Scope / Non-Goals + +- Vector/embedding search on artifacts (deferred — schema supports future extension) +- DB export/dump command +- Changing file discovery in deriveState (stays on disk) +- Making createWorktree async (keep sync, use copyFileSync for DB copy) + +## Technical Constraints + +- `node:sqlite` is experimental — use DbAdapter abstraction to insulate +- `node:sqlite` returns null-prototype rows — normalize via spread in DbAdapter +- Named SQL parameters must use colon-prefix (`:id`, `:scope`) for `node:sqlite` compatibility +- `createWorktree` must remain synchronous — no async cascade +- All DB operations must be wrapped in try/catch with fallback to existing behavior +- Memory-db source code is reference — adapt to current architecture, don't copy blindly + +## Integration Points + +- `auto-prompts.ts` — replace `inlineGsdRootFile` with DB-aware helpers (scoped queries with filesystem fallback) +- `auto.ts` `startAuto()` — DB open + auto-migration before first dispatch +- `auto.ts` `handleAgentEnd()` — re-import markdown after auto-commit (after doctor + rebuildState, before dispatch) +- `metrics.ts` — extend `UnitMetrics` with measurement fields, extend `snapshotUnitMetrics` signature +- `state.ts` `deriveState()` — DB-first content loading with filesystem fallback +- `worktree-manager.ts` `createWorktree()` — sync DB copy after worktree creation +- `worktree-command.ts` / merge paths — DB reconciliation after merge +- `index.ts` — 3 new tool registrations +- `commands.ts` — `/gsd inspect` command registration +- `types.ts` — Decision/Requirement interface additions + +## Open Questions + +- Whether memory-db's custom DECISIONS.md parser handles the current format (pipe tables with supersession chains) — needs verification during S02 implementation +- Whether current `deriveState()` batch-parse logic is structurally compatible with the DB-first replacement — needs verification during S04 diff --git a/.gsd/milestones/M004/M004-ROADMAP.md b/.gsd/milestones/M004/M004-ROADMAP.md new file mode 100644 index 000000000..73fce2281 --- /dev/null +++ b/.gsd/milestones/M004/M004-ROADMAP.md @@ -0,0 +1,197 @@ +# M004: SQLite Context Store — Surgical Prompt Injection + +**Vision:** Replace GSD's whole-file prompt dumps with a SQLite-backed query layer that surgically injects only the context each dispatch unit needs — delivering ≥30% token savings, eliminating context pollution, and enabling structured LLM output that bypasses fragile markdown parsing. + +## Success Criteria + +- All prompt builders use DB queries for context injection (zero direct `inlineGsdRootFile` for data artifacts in prompt builders) +- Existing GSD projects migrate silently to DB on first run with zero data loss +- Planning and research dispatch units show ≥30% fewer prompt characters on mature projects +- System works identically via fallback when SQLite unavailable — no crash, transparent degradation +- Worktree creation copies gsd.db; worktree merge reconciles rows +- LLM can write decisions/requirements/summaries via structured tool calls +- `/gsd inspect` shows DB state for debugging +- Dual-write keeps markdown files in sync with DB state in both directions +- `deriveState()` reads from DB when available, falls back to filesystem +- All existing tests continue to pass, TypeScript compiles clean + +## Key Risks / Unknowns + +- `auto-prompts.ts` has 11 prompt builders with 19 `inlineGsdRootFile` calls — rewiring is high-surface-area +- `handleAgentEnd` has new post-unit-hook/doctor/rebuildState machinery — dual-write re-import must integrate cleanly +- Memory-db's custom markdown parsers may not handle format changes since the fork point +- `node:sqlite` is experimental — API stability risk (mitigated by DbAdapter abstraction) + +## Proof Strategy + +- SQLite provider risk → retire in S01 by proving tiered chain loads and queries on target platform +- Parser/format risk → retire in S02 by round-trip testing every artifact type against current file formats +- Prompt builder rewiring risk → retire in S03 by verifying all 11 builders produce correct output with DB vs markdown +- Worktree integration risk → retire in S05 by testing copy/reconcile against current worktree architecture + +## Verification Classes + +- Contract verification: unit tests for DB layer, importers, query layer, state derivation, writer, tools. Round-trip fidelity tests for migration. +- Integration verification: prompt builders produce equivalent output with DB vs markdown. Full auto-mode cycle completes. Worktree DB copy/merge works. +- Operational verification: graceful fallback when SQLite unavailable. Token measurement reports savings ≥30%. +- UAT / human verification: user runs auto-mode on a real project and confirms output quality equivalent or better + +## Milestone Definition of Done + +This milestone is complete only when all are true: + +- All prompt builders in `auto-prompts.ts` use DB queries for context injection +- Silent auto-migration works on existing GSD projects with all artifact types +- Dual-write keeps markdown files in sync with DB state (both directions) +- Graceful fallback to markdown when SQLite unavailable +- Token measurement shows ≥30% reduction on planning/research units +- `deriveState()` derives from DB, producing identical GSDState output +- Worktree DB copy and merge reconciliation work with current worktree architecture +- Structured LLM tools registered and functional with DB-first write +- `/gsd inspect` command works +- All existing tests pass, new DB test suite passes, `npx tsc --noEmit` clean +- Success criteria re-checked against live behavior + +## Requirement Coverage + +- Covers: R045, R046, R047, R048, R049, R050, R051, R052, R053, R054, R055, R056, R057 +- Partially covers: none +- Leaves for later: none +- Orphan risks: none + +## Slices + +- [ ] **S01: DB Foundation + Schema** `risk:high` `depends:[]` + > After this: SQLite DB opens with tiered provider chain, schema inits with decisions/requirements/artifacts tables plus filtered views, typed CRUD wrappers work, graceful fallback returns empty results when SQLite unavailable. Proven by unit tests against real DB. + +- [ ] **S02: Markdown Importers + Auto-Migration** `risk:medium` `depends:[S01]` + > After this: Existing GSD project with markdown files starts up → gsd.db appears silently with all artifact types imported. Round-trip fidelity proven for every artifact type — import then regenerate produces identical output. + +- [ ] **S03: Surgical Prompt Injection + Dual-Write** `risk:high` `depends:[S01,S02]` + > After this: All 11 `build*Prompt()` functions in `auto-prompts.ts` use scoped DB queries instead of `inlineGsdRootFile`. Decisions filtered by milestone, requirements filtered by slice. Dual-write re-import in `handleAgentEnd` keeps DB in sync after each dispatch unit. Falls back to filesystem when DB unavailable. + +- [ ] **S04: Token Measurement + State Derivation** `risk:medium` `depends:[S03]` + > After this: `promptCharCount`/`baselineCharCount` in UnitMetrics, measurement wired into all `snapshotUnitMetrics` call sites. `deriveState()` reads content from DB when available. Savings ≥30% confirmed on fixture data. + +- [ ] **S05: Worktree DB Isolation** `risk:medium` `depends:[S01,S02]` + > After this: `createWorktree` copies gsd.db to new worktrees (sync, non-fatal). Merge paths reconcile worktree DB rows back via ATTACH DATABASE with conflict detection. + +- [ ] **S06: Structured LLM Tools + /gsd inspect** `risk:medium` `depends:[S03]` + > After this: LLM writes decisions/requirements/summaries via tool calls that write to DB first, then regenerate markdown. `/gsd inspect` dumps schema version, table counts, recent entries. + +- [ ] **S07: Integration Verification + Polish** `risk:low` `depends:[S03,S04,S05,S06]` + > After this: Full auto-mode lifecycle test proves all subsystems compose correctly — migration → scoped queries → formatted prompts → token savings → re-import → round-trip. Edge cases (empty projects, partial migrations, fallback mode) verified. ≥30% savings confirmed on realistic fixture data. + +## Boundary Map + +### S01 → S02 + +Produces: +- `gsd-db.ts` → `openDatabase()`, `closeDatabase()`, `initSchema()`, `migrateSchema()`, typed insert/query wrappers for decisions, requirements, artifacts tables +- `gsd-db.ts` → `isDbAvailable()` boolean, `getDbProvider()` provider name +- `gsd-db.ts` → `insertDecision()`, `insertRequirement()`, `insertArtifact()`, `upsertDecision()`, `upsertRequirement()` +- `gsd-db.ts` → `transaction()` wrapper for batch operations +- `context-store.ts` → `queryDecisions(opts?)`, `queryRequirements(opts?)`, `queryArtifact(path)`, `queryProject()` +- `context-store.ts` → `formatDecisionsForPrompt()`, `formatRequirementsForPrompt()` +- `types.ts` → `Decision`, `Requirement` interfaces +- Fallback: all query functions return empty when DB unavailable + +Consumes: +- nothing (first slice) + +### S01 → S03 + +Produces: +- Same as S01 → S02 (DB layer + query functions + formatters) +- `isDbAvailable()` for conditional DB vs markdown loading in prompt builders + +Consumes: +- nothing (first slice) + +### S01 → S05 + +Produces: +- `gsd-db.ts` → `copyWorktreeDb(srcPath, destPath)` — sync file copy +- `gsd-db.ts` → `reconcileWorktreeDb(mainDbPath, worktreeDbPath)` — ATTACH-based merge +- `openDatabase()` for opening DB at arbitrary paths + +Consumes: +- nothing (first slice) + +### S02 → S03 + +Produces: +- `md-importer.ts` → `migrateFromMarkdown(basePath)` — full project import function +- `md-importer.ts` → individual parsers for all artifact types +- Auto-migration detection and execution wired into `startAuto()` + +Consumes from S01: +- `gsd-db.ts` → `openDatabase()`, typed insert wrappers, `transaction()` +- Schema tables for all artifact types + +### S02 → S05 + +Produces: +- `md-importer.ts` → `migrateFromMarkdown()` for importing markdown into a fresh worktree DB + +Consumes from S01: +- `gsd-db.ts` → database layer + +### S03 → S04 + +Produces: +- All `build*Prompt()` functions rewired to use DB queries +- DB-aware inline helpers: `inlineDecisionsFromDb()`, `inlineRequirementsFromDb()`, `inlineProjectFromDb()` +- Dual-write re-import in `handleAgentEnd` + +Consumes from S01: +- `context-store.ts` → query functions and formatters +- `gsd-db.ts` → `isDbAvailable()` + +Consumes from S02: +- `md-importer.ts` → `migrateFromMarkdown()` for re-import after auto-commit + +### S03 → S06 + +Produces: +- `context-store.ts` → complete query layer that structured tools can use +- Dual-write infrastructure (re-import pattern) + +Consumes from S01: +- `gsd-db.ts` → typed upsert wrappers + +### S04 → S07 + +Produces: +- Token measurement in `UnitMetrics` (`promptCharCount`, `baselineCharCount`) +- `deriveState()` DB-first content loading +- Measurement infrastructure in `dispatchNextUnit` + +Consumes from S03: +- Rewired prompt builders + +### S05 → S07 + +Produces: +- `copyWorktreeDb` wired into `createWorktree` +- `reconcileWorktreeDb` wired into merge paths + +Consumes from S01: +- `gsd-db.ts` → `copyWorktreeDb()`, `reconcileWorktreeDb()`, `openDatabase()` + +Consumes from S02: +- `md-importer.ts` → `migrateFromMarkdown()` for fallback import + +### S06 → S07 + +Produces: +- 3 structured LLM tools registered: `gsd_save_decision`, `gsd_update_requirement`, `gsd_save_summary` +- `/gsd inspect` slash command with autocomplete + +Consumes from S03: +- `context-store.ts` → query layer for inspect output +- Dual-write infrastructure for tool-triggered markdown regeneration + +Consumes from S01: +- `gsd-db.ts` → `upsertDecision()`, `upsertRequirement()`, `insertArtifact()` +- `db-writer.ts` → `generateDecisionsMd()`, `generateRequirementsMd()`, DB-first write helpers