# Proposal: DB-First Planning State — Markdown as Render Target **Status:** Draft **Date:** 2026-05-15 **Author:** Autonomous session + design review **Relates to:** `.sf/PRINCIPLES.md` L1, `docs/adr/0000-purpose-to-software-compiler.md` --- ## Executive Summary SF's principle states "SQLite is the canonical structured store … treat `.sf/sf.db` as the first place for planning hierarchy." The implementation has not followed this: milestone artifact files (`CONTEXT.md`, `ROADMAP.md`, `VALIDATION.md`, `SUMMARY.md`, and their slice/task counterparts) are written and edited as primary sources of truth, with DB rows either reconciled after the fact or only partially kept in sync. The result is observable drift that costs the autonomous orchestrator agent turns and derails long runs. This proposal describes how to complete the inversion: DB rows become the canonical source, markdown files become a deterministic render of that state, and the doctor/reconciler logic collapses to near-zero because there is nothing to reconcile. --- ## 1. Current State Survey ### 1.1 Artifact Inventory The table below covers the artifact types found under `.sf/milestones/M*/` (surveyed against M001-6377a4 as the reference case, confirmed by `grep` across the extension source). | Artifact | Example path | Written by | Read by | DB representation today | Authoritative today | |---|---|---|---|---|---| | `M*-CONTEXT.md` | `M001-6377a4/M001-6377a4-CONTEXT.md` | Bootstrap agent via `write_file`; `record-promoter.js` (migration path); `migrate/writer.js` | `state-db.js` `buildRegistryAndFindActive` (title extraction); `reasoning-assist.js` (injects into prompts); `auto-start.js` (presence check); `auto-direct-dispatch.js` | Partial: `milestones.vision`, `vision_meeting_json`, `product_research_json`, `boundary_map_markdown`, `requirement_coverage` hold structured fragments; the full narrative prose has no DB column | **File** — title is extracted from prose; structured fields exist but are not always populated | | `M*-CONTEXT-DRAFT.md` | `M001-6377a4/M001-6377a4-CONTEXT-DRAFT.md` | Bootstrap agent via `write_file`; `triage-resolution.js` | `state-db.js` (presence check: draft → active milestone not yet planned); `auto-start.js` | None | **File** — only signal is presence/absence | | `M*-ROADMAP.md` | `M001-6377a4/M001-6377a4-ROADMAP.md` | `renderRoadmapFromDb` in `markdown-renderer.js`; `record-promoter.js`; `migrate/writer.js` | `state-db.js` (fallback title when DB title blank); `doctor-engine-checks.js` (`detectMissingPlanningRowsFromProjection`); `post-execution-checks.js` (`checkMilestoneIntegrity`); `commands-ship.js`; `commands-handlers.js` | DB: `milestones` + `slices` tables hold all slice hierarchy, sequence, status, depends | **Ambiguous** — DB is authoritative for status/hierarchy; ROADMAP.md is a render target in theory but is also the fallback when DB is empty | | `M*-ROADMAP.json` | `M001-6377a4/M001-6377a4-ROADMAP.json` | `writeRoadmapJsonProjection` (called from `renderRoadmapFromDb`) | `doctor-engine-checks.js` (`projectionDriftIssues`) — read-only drift check | DB: same `milestones`/`slices` tables | **Projection** — correctly labelled as "export/recovery only; DB remains authoritative" in doctor warning | | `M*-VALIDATION.md` | `M001-6377a4/M001-6377a4-VALIDATION.md` | `handleValidateMilestone` in `tools/validate-milestone.js` (DB write first, then disk render) | `state-db.js` `readMilestoneValidationVerdict` (terminal check); `doctor-engine-checks.js` `projectionDriftIssues`; `workspace-index.js` (`validationVerdict` field); `auto-prompts.js` (injected as context); `reflection.js`; `complete-milestone.js` (re-reads verdict from file); `auto-worktree.js` | DB: `assessments` table (full_content, status/verdict, scope=milestone-validation) | **Ambiguous** — `handleValidateMilestone` writes DB first, then file; but `complete-milestone.js` re-reads the file, and `workspace-index.js` reads the file, not the DB | | `M*-SUMMARY.md` | (milestone-level, e.g. `M001-6377a4-SUMMARY.md`) | `handleCompleteMilestone` in `tools/complete-milestone.js` | `state-db.js` `buildRegistryAndFindActive` (title fallback for complete milestones) | DB: `milestone_evidence` (narrative rendered) + milestone `status/completed_at`; no `full_summary_md` column on milestones table | **File** — absence of SUMMARY.md can block the `completing-milestone` phase even when DB says complete | | `S*-SUMMARY.md` | `slices/S01/S01-SUMMARY.md` | Agent via `write_file` (autonomous solver), then `renderSliceSummary` backfills from DB `slices.full_summary_md` | `summary-helpers.js` (prior task context injection); doctor completeness check | DB: `slices.full_summary_md` TEXT column | **Ambiguous** — file written by agent first, DB column populated by `setSliceSummaryMd`; renderer then writes back from DB | | `S*-UAT.md` | `slices/S01/S01-UAT.md` | Agent via `write_file`; `renderSliceSummary` (writes from `slices.full_uat_md`) | Doctor completeness check | DB: `slices.full_uat_md` TEXT column; `slices.uat_verdict` | **Ambiguous** — same dual-write pattern as SUMMARY.md | | `S*/tasks/T*-SUMMARY.md` | `slices/S01/tasks/T01-SUMMARY.md` | Agent via `write_file`; `renderTaskSummary` (from `tasks.full_summary_md`) | `summary-helpers.js` prior-task context; doctor | DB: `tasks.full_summary_md` TEXT column | **Ambiguous** — task summary lives in DB but file is what the agent reads next turn | | `S*-PLAN.md` / `T*-PLAN.md` | `slices/S03/S03-PLAN.md` | `renderSlicePlanFromDb` / `renderTaskPlanFromDb` in `markdown-renderer.js` | `post-execution-checks.js`; agent via `read_file` | DB: `slices.goal`, `success_criteria`, `full_summary_md`; `tasks.full_plan_md`, `description`, etc. | **DB-first in theory** — renderers exist; but slice PLAN is sometimes skipped if `full_plan_md` is empty | | `M*-CONTEXT-DRAFT.md` (anchor) | `anchors/plan-slice.json` | `plan` tooling | `auto-dispatch.js` (planning anchor gate) | DB: none | **File** | ### 1.2 Key Readers of Each Artifact **Who reads `VALIDATION.md`:** - `state-db.js:readMilestoneValidationVerdict` — terminal verdict check; blocks completing the milestone if not terminal - `workspace-index.js:buildWorkspaceIndex` — populates `validationVerdict` on each milestone entry (used by TUI, context injection) - `auto-prompts.js:buildValidateMilestonePrompt` — reads file as context block for re-validation rounds - `complete-milestone.js:handleCompleteMilestone` — re-reads file verdict as guard before marking complete - `reflection.js:readValidationFiles` — reads all milestone VALIDATION files for the reflection corpus - `doctor-engine-checks.js:projectionDriftIssues` — compares file verdict vs DB `assessments.status` **Who reads `CONTEXT.md`:** - `state-db.js:buildRegistryAndFindActive` — title extraction fallback when `milestones.title` is blank - `reasoning-assist.js` — injects context into prompt blocks - `auto-start.js` — presence check (CONTEXT exists → bootstrap complete) - `auto-direct-dispatch.js` — reads slice CONTEXT for dispatch context **Who reads `ROADMAP.md`:** - `doctor-engine-checks.js:detectMissingPlanningRowsFromProjection` — if ROADMAP.md has slices but DB has none, warns - `post-execution-checks.js:checkMilestoneIntegrity` — reads ROADMAP.md to verify done slices have SUMMARY.md - `commands-ship.js` — reads roadmap for ship-gate check - `state-db.js:534` — checks whether ROADMAP file exists as a proxy for "milestone has been planned" **Who reads `SUMMARY.md` (slice/task):** - `summary-helpers.js` — injected as prior-task context into execute-task prompts - `state-db.js:buildRegistryAndFindActive` — reads milestone SUMMARY for title (complete milestone fallback) - `markdown-renderer.js:detectStaleRenders` — checks presence of SUMMARY.md for complete tasks ### 1.3 Authoritative-Today Summary ``` Artifact | Source of truth today | Drift risk --------------------|--------------------------|------------------ CONTEXT.md | File (prose) | High — title field in DB often blank CONTEXT-DRAFT.md | File (presence signal) | Low (no content read) ROADMAP.md | Ambiguous — DB for status, file for completeness check | Medium ROADMAP.json | DB (projection) | Low — doctor already catches drift VALIDATION.md | File (re-read by complete-milestone) | High — this session's critical failure SUMMARY.md (ms) | File (blocking) | High — absence caused stuck phase SUMMARY.md (slice) | Ambiguous (dual-write) | Medium UAT.md (slice) | Ambiguous (dual-write) | Medium SUMMARY.md (task) | Ambiguous (dual-write) | Medium PLAN.md (slice/task)| DB-first via renderers | Low (renderer exists) ``` --- ## 2. Proposed Model ### 2.1 Canonical DB Representation (Reusing Existing Tables) The existing schema already holds or can hold the authoritative state for every artifact type listed above. No new tables are required for the MVP; a few additive column additions are needed for the full rollout. #### VALIDATION.md → `assessments` table Already done (partially). `handleValidateMilestone` already writes `assessments` (full_content, status, scope=milestone-validation) before writing the file. The gap is that three consumers re-read the file instead of querying the DB. **No schema change required.** #### ROADMAP.md → `milestones` + `slices` tables Already fully represented. Every slice's id, title, status, sequence, risk, depends, and done state are in the `slices` table. The `renderRoadmapFromDb` function in `markdown-renderer.js` already generates correct markdown from these rows. **No schema change required.** #### SUMMARY.md (milestone) → `milestones` + `milestone_evidence` tables The milestone `complete-milestone` handler writes narrative to `milestone_evidence` but there is no `milestones.full_summary_md` column. Adding one mirrors the pattern used by `slices.full_summary_md` and `tasks.full_summary_md`. **Additive schema change required:** ```sql ALTER TABLE milestones ADD COLUMN full_summary_md TEXT NOT NULL DEFAULT ''; ``` #### SUMMARY.md (slice) → `slices.full_summary_md` Already exists. `setSliceSummaryMd` writes this column. `renderSliceSummary` reads it and writes to disk. **No schema change required.** #### UAT.md (slice) → `slices.full_uat_md` Already exists. Same pattern as SUMMARY.md. **No schema change required.** #### CONTEXT.md → `milestones` table (structured fields) + new `full_context_md` column The `milestones` table already holds structured fields extracted from CONTEXT.md (`vision`, `success_criteria`, `key_risks`, `proof_strategy`, `verification_contract/integration/operational/uat`, `definition_of_done`, `requirement_coverage`, `boundary_map_markdown`). What is missing is a column for the full human-authored narrative prose (the "Project Description", "Why This Milestone", "User-Visible Outcome" sections). Adding `full_context_md TEXT NOT NULL DEFAULT ''` allows the complete context to be stored and rendered on demand. **Additive schema change required:** ```sql ALTER TABLE milestones ADD COLUMN full_context_md TEXT NOT NULL DEFAULT ''; ``` #### CONTEXT-DRAFT.md → `milestones.status` + new `context_draft_md` column The draft is a signal that planning is in progress. The presence check can be replaced by a column that stores the draft content; status `queued` with non-empty `context_draft_md` replaces the file presence check. **Additive schema change required:** ```sql ALTER TABLE milestones ADD COLUMN context_draft_md TEXT NOT NULL DEFAULT ''; ``` ### 2.2 Markdown as a Projection: Lazy vs Eager **Decision: Eager generation on write, with lazy fallback on read.** The render target must be written to disk at the same transaction boundary as the DB write, for two reasons: 1. **Git atomicity**: human reviewers see PR diffs. If the markdown file is absent or stale, the diff is hard to review. Committing DB state without a corresponding markdown update breaks review UX. 2. **Existing readers that read the file**: `complete-milestone.js`, `workspace-index.js`, `reflection.js`, `auto-prompts.js` all read the file path directly. Migrating all call sites at once is a large change; eager disk writes allow incremental migration. Lazy generation (render only when `cat` or `sf plan show` is called) is appealing in the steady state but breaks the git-atomicity requirement and existing consumers during the transition period. A lazy-only model is the long-term ideal (addressed in Section 7), but not safe for the MVP. **The projection contract:** ``` DB write → render markdown → atomic disk write → git commit ↑ deterministic function; same DB state → same markdown content every time ``` The rendered file gets a comment header: ```markdown ``` For CONTEXT.md and CONTEXT-DRAFT.md, the "do not edit" header is softer because they contain human-authored prose that has no other editing path during the bootstrap phase. Editing UX is addressed in Section 3. ### 2.3 Migration: Existing Markdown → DB Import For each artifact type, the migration follows this pattern: 1. Read the existing markdown file. 2. Parse the structured content using existing parsers (`parseRoadmap`, `extractVerdict`, `parseSummary`, frontmatter parser). 3. Upsert the parsed content into the appropriate DB column(s) using existing functions (e.g., `upsertMilestonePlanning`, `setSliceSummaryMd`, `insertAssessment`). 4. Re-render the file from DB (which adds the `` header and normalizes formatting). 5. If the re-rendered content differs from the original, commit as a single atomic operation. **One-time migration function location:** `src/resources/extensions/sf/migrate/db-first-import.js`. It is called once by `doctor --fix` when the `db_markdown_not_imported` doctor code fires; never called on every startup. The migration is idempotent: if `assessments.full_content` already matches the file's content hash, it is a no-op. --- ## 3. Editing UX ### 3.1 Options **(a) Edit markdown → parse back into DB (round-trip-safe)** Requires a robust bidirectional parser for every artifact type. CONTEXT.md is free-form prose; there is no safe round-trip parser. Any ambiguous or model-generated prose in CONTEXT.md would be lost, truncated, or corrupted on re-import. High implementation cost; fragile. **(b) Markdown is read-only generated; edits go through `sf plan` commands or `sf headless db ...`** Clean separation. Agents and operators must use structured commands to mutate state. The markdown file is authoritative only for reading/reviewing. **(c) Hybrid: drafts are markdown, promotion to canonical writes DB and regenerates markdown** CONTEXT-DRAFT.md remains agent-editable (it is the active working surface during bootstrap). Once `plan_milestone` is called, CONTEXT.md is generated from DB and becomes read-only. **Decision: Option (c) — hybrid.** Reasoning: - CONTEXT-DRAFT.md and slice/task PLAN.md are the primary LLM write surfaces during active work. Making them DB-only would require adding LLM tool calls for every sentence of the planning meeting. The bootstrap and planning flows are already LLM-driven with free-form file writes. - VALIDATION.md, ROADMAP.md, SUMMARY.md, and UAT.md are outputs of structured operations (`validate_milestone`, `plan_milestone`, `complete_slice`, `complete_task`). These have clean tool-call boundaries and can safely become DB-first with no editing path for the file itself. - The hybrid lets us complete the MVP without restructuring the bootstrap/planning LLM flow, while still eliminating the drift that caused the session failures. **Editing rules under the hybrid model:** | Artifact | Editing path | File state | |---|---|---| | CONTEXT-DRAFT.md | Agent writes via `write_file` during bootstrap; content stored in `milestones.context_draft_md` on save | Editable (agent/human) | | CONTEXT.md | `plan_milestone` call generates from DB fields + draft content; file becomes read-only | `` header | | ROADMAP.md | `plan_milestone` / `sf plan update-slice` commands; file is regenerated | `` header | | ROADMAP.json | Regenerated alongside ROADMAP.md | `// generated` comment | | VALIDATION.md | `validate_milestone` tool call; file regenerated from DB | `` header | | SUMMARY.md (slice/task) | `complete_slice` / `complete_task` tool calls; field written to DB, file generated | `` header | | UAT.md | `complete_slice` tool call (uat_md param); field written to DB | `` header | | SUMMARY.md (milestone) | `complete_milestone` tool call; milestone `full_summary_md` column written, file generated | `` header | For operators who need to edit a generated file (e.g., to correct a validation rationale), the path is: 1. `sf headless db -- "UPDATE assessments SET full_content = '...' WHERE milestone_id = 'M001'"` — or via `sf plan validate --update-rationale`. 2. The update triggers a projection re-render (file is rewritten from DB). --- ## 4. Stale-Detection and Reconciliation ### 4.1 Current Reconciliation Cost Today, `doctor-engine-checks.js:projectionDriftIssues` performs two checks: 1. **ROADMAP.json drift**: compares DB slice order/status against JSON file. Fires `db_projection_roadmap_drift` warning. 2. **VALIDATION.md verdict drift**: compares DB `assessments.status` against file frontmatter `verdict`. Fires `db_projection_validation_drift` warning. This is the check that fired during the M001-6377a4 session ("verdict in markdown said needs-remediation but DB said needs-attention"). Additionally, `detectStaleRenders` in `markdown-renderer.js` checks checkbox states and missing SUMMARY.md files. `workspace-index.js` reads VALIDATION files independently of the DB. `state-db.js` reads CONTEXT.md for title extraction when the DB title is blank. Each of these is a potential source of drift. The autonomous orchestrator spent turns reasoning about the discrepancy before acting on it. ### 4.2 After DB-First Migration: Doctor Changes **For `VALIDATION.md`:** The `db_projection_validation_drift` check is eliminated entirely. The file is always regenerated from DB by `handleValidateMilestone`. The doctor check becomes: ``` if (validationPath exists AND full_content in assessments is empty) { issue: db_validation_content_missing — file exists but DB has no stored content; run `sf doctor --fix` to import the file into DB. severity: info, fixable: true } ``` This inverts the check: instead of asking "does the file match the DB?", it asks "is the DB populated for this file?". Once the one-time import runs, the check never fires again. **For `ROADMAP.json`:** The `db_projection_roadmap_drift` check remains as a sanity assertion during the transition period. After full rollout, it is removed: the JSON file is always regenerated alongside ROADMAP.md by `renderRoadmapFromDb`. **For `SUMMARY.md` / `UAT.md` missing files:** `detectStaleRenders` currently detects when a complete task/slice has content in DB but no file on disk. After DB-first, the render is triggered eagerly by `complete_task`/`complete_slice`, so the file is never missing. The check becomes a recovery path only (fired when the disk write failed mid-way). **For `CONTEXT.md` title fallback:** `state-db.js:buildRegistryAndFindActive` falls back to reading CONTEXT.md when `milestones.title` is blank. After DB-first, the `full_context_md` column is populated on every `plan_milestone` call, and `milestones.title` is always written alongside it. The file-system fallback path is removed from `buildRegistryAndFindActive`. **For `CONTEXT.md` presence check in `auto-start.js`:** The check `if (!contextFile && draftFile) activeMilestoneHasDraft = true` is replaced by checking `milestones.context_draft_md IS NOT NULL AND context_draft_md != ''`. **Net result:** the doctor has nothing to reconcile for these artifact types. The autonomous orchestrator spends zero turns on drift detection. The estimated ~20% reconciliation overhead in early autonomous runs goes to near-zero. --- ## 5. Migration Plan ### 5.1 MVP: `*-VALIDATION.md` **Why first:** - Smallest file (frontmatter + ~30 lines); parser (`extractVerdict`) already exists and is battle-tested. - The doctor already detects drift (`db_projection_validation_drift`), providing a clear before/after signal. - `handleValidateMilestone` already writes DB before disk — the pattern is already correct at the write site. The remaining work is fixing the three read sites. - The session failure was specifically caused by stale VALIDATION.md verdict. This is the highest-pain point. **Step-by-step for VALIDATION.md MVP:** **Step V1: Fix read sites to prefer DB.** Files to change: - `complete-milestone.js:handleCompleteMilestone` — line 136-142: replace `extractVerdict(readFileSync(validationPath))` with `getMilestoneValidationAssessment(milestoneId)?.status`. - `workspace-index.js:buildWorkspaceIndex` — replace file-read loop with `getMilestoneValidationAssessment(milestone.id)?.status`. - `state-db.js:readMilestoneValidationVerdict` — replace file-read with `getMilestoneValidationAssessment(milestoneId)?.status`; keep file-read as fallback only when DB row is absent (for repos that have not yet run the migration). Rollback: trivially revert the three call-site changes. No schema change, no data loss. Readers that do NOT switch to DB in this step: - `auto-prompts.js:buildValidateMilestonePrompt` — still reads the file as a human-readable context block (acceptable; the file is generated from DB so it will be correct). - `reflection.js:readValidationFiles` — still reads files for reflection corpus (acceptable; same reason). **Tests:** - Add a test in `src/resources/extensions/sf/tests/` that calls `handleValidateMilestone`, then checks that `getMilestoneValidationAssessment` returns the verdict — and that the same verdict is returned by `readMilestoneValidationVerdict` without any file on disk. - Extend `doctor-engine-checks` tests to confirm that `db_projection_validation_drift` no longer fires for a freshly validated milestone. **Step V2: Add `` header to VALIDATION.md output.** In `renderValidationMarkdown` inside `tools/validate-milestone.js`, prepend: ```markdown ``` Update `verdict-parser.js:extractVerdict` to skip this comment line when parsing verdicts from files. **Tests:** confirm that `extractVerdict` still works on files with the header; confirm that the doctor check does not false-positive on the header. **Step V3: One-time import doctor fix.** Add doctor code `db_validation_content_missing`: fires when a `VALIDATION.md` file exists for a milestone but `assessments` has no row for `scope=milestone-validation` for that milestone. Doctor `--fix` calls `insertAssessment` with the file content. **Tests:** set up a milestone with a VALIDATION.md but no DB row; confirm doctor fires and fix imports it. **Step V4: Remove `db_projection_validation_drift` doctor check.** Once V1–V3 are in and the migration has run, the drift check is provably unreachable (the file is always generated from DB). Remove it to reduce noise. **Rollback at any step:** each step is independently revertible. The DB `assessments` rows are not deleted during any step, so reverting the read-site changes restores the file-read behavior. --- ### 5.2 Incremental Rollout to CONTEXT.md **Dependencies:** CONTEXT.md is not currently in the DB at the full-prose level. The `full_context_md` schema addition must land first. **Sequence:** 1. Add `milestones.full_context_md TEXT NOT NULL DEFAULT ''` migration (schema version bump in `sf-db-schema.js`). 2. Update `plan_milestone` tool handler to write the full context prose to `full_context_md` in addition to the existing structured fields. 3. Update `state-db.js:buildRegistryAndFindActive` title extraction to read from DB title first (already done for non-empty titles); add fallback to extract title from `full_context_md` before falling back to file. 4. Update `triage-resolution.js` CONTEXT-DRAFT.md seed to also write `milestones.context_draft_md`. 5. Add `renderContextFromDb` function to `markdown-renderer.js` that generates CONTEXT.md from `full_context_md` (or structured fields if `full_context_md` is empty). 6. Update `plan_milestone` to call `renderContextFromDb` after writing DB, replacing the direct `write_file` instruction in the LLM prompt for the CONTEXT.md write. 7. Doctor check: `db_context_missing` fires when CONTEXT.md exists but `full_context_md` is empty; `--fix` imports file content. 8. CONTEXT-DRAFT.md: `auto-start.js` presence check replaced by `milestones.context_draft_md IS NOT NULL`. **Tests:** planning flow integration test that verifies `plan_milestone` writes `full_context_md` to DB and regenerates CONTEXT.md with correct content and header. --- ### 5.3 Incremental Rollout to ROADMAP.md ROADMAP.md is already partially DB-first: `renderRoadmapFromDb` exists and is called by the slice/task completion path. The remaining work is: 1. Ensure `plan_milestone` always calls `renderRoadmapFromDb` instead of (or in addition to) writing ROADMAP.md via direct file instruction. 2. Replace `state-db.js:534` file-existence proxy (`resolveMilestoneFile(basePath, activeMilestone.id, 'ROADMAP') !== null`) with a DB check (`getMilestoneSlices(milestoneId).length > 0`). 3. Replace `post-execution-checks.js:checkMilestoneIntegrity` ROADMAP read with a direct DB query for done slices, then check SUMMARY.md presence separately. 4. Add `` header; update `parseRoadmap` to skip the header line. 5. Doctor check: `db_roadmap_not_imported` fires when ROADMAP.md has slices not in DB; `--fix` calls `migrateHierarchyToDb` (already exists in `md-importer.js`). --- ### 5.4 Incremental Rollout to SUMMARY.md (Milestone) 1. Add `milestones.full_summary_md TEXT NOT NULL DEFAULT ''` schema migration. 2. Update `handleCompleteMilestone` to write `full_summary_md` to the milestones table. 3. Add `renderMilestoneSummaryFromDb` to `markdown-renderer.js`. 4. Call the renderer from `handleCompleteMilestone` (replacing the direct `saveFile` call or running alongside it during transition). 5. Update `state-db.js:buildRegistryAndFindActive` title fallback for complete milestones to read from `milestones.full_summary_md` (or `milestones.title`) before reading the file. 6. This eliminates the "SUMMARY.md missing → completing-milestone phase stuck" failure mode: the DB row is written first, and the render is a deterministic step. --- ### 5.5 Rollout Order Summary | Phase | Artifact | Required schema change | Estimated effort | |---|---|---|---| | MVP | VALIDATION.md | None | Small (3 read-site changes + tests) | | 2 | SUMMARY.md (slice, task) | None (columns exist) | Small (add header, fix stale-render fallback) | | 3 | ROADMAP.md | None | Medium (dispatch-side guard change + test) | | 4 | SUMMARY.md (milestone) | `milestones.full_summary_md` | Medium | | 5 | CONTEXT.md | `milestones.full_context_md` | Large (LLM flow change) | | 6 | CONTEXT-DRAFT.md | `milestones.context_draft_md` | Small (presence check replacement) | --- ## 6. Non-Goals - **Does NOT replace `.sf/journal/*.jsonl`** (append-only event log; complementary to DB; no change). - **Does NOT replace ADRs / proposals in `docs/`** (human-authored; promoted-only; not generated). See PRINCIPLES.md L4: "promote only reviewed plans, specs, and ADRs to `docs/`". - **Does NOT abolish markdown.** Markdown files under `.sf/milestones/` remain on disk and are committed to git. They are readable by humans and LLMs. The only change is that they are generated artifacts rather than primary state. The git history of these files remains meaningful as a timeline of validation and completion events. - **Does NOT change the slice PLAN.md or task PLAN.md write flow.** These are already DB-first (via `renderSlicePlanFromDb` / `renderTaskPlanFromDb`). The `full_plan_md` column exists on `tasks`. No migration needed. - **Does NOT change `.sf/runtime/` files** (solver checkpoint, unit runtime record). These are ephemeral operational state, not planning state. - **Does NOT change `docs/records/` workflow.** The `record-promoter.js` promotes human-authored records to milestone shells; those shells still use CONTEXT-DRAFT.md as the initial planning surface. - **Does NOT gate on a single DB technology.** The proposal assumes SQLite via `sf-db.js`. No change to the database engine. --- ## 7. Open Questions ### Q1: How are DB-stored rich-text fields exported to markdown (rendering rules)? The `renderValidationMarkdown` and `renderRoadmapMarkdown` functions in `markdown-renderer.js` and `validate-milestone.js` already implement this for their respective artifact types. For CONTEXT.md, the rich-text is free-form prose stored verbatim in `full_context_md`; the renderer wraps it in the standard header/metadata block. No special encoding or schema migration is needed for plain markdown prose fields. For structured list fields (e.g., `milestones.success_criteria` stored as JSON array), the existing `renderListEntry` helper in `markdown-renderer.js` handles the serialization. The schema migration path (adding a column) is gated on the existing `columnExists` helper and the versioned migration table in `sf-db-schema.js`. **Unresolved:** What happens when the free-form prose in `full_context_md` contains markdown that conflicts with the generated header/section structure? The current `normalizeMarkdownBlockSpacing` helper handles some cases, but adversarial LLM-authored content with duplicate H1 headings or frontmatter blocks could corrupt the render. A content sanitizer for known problematic patterns should be added to the render pipeline before Phase 5 (CONTEXT.md). ### Q2: Should there be a "view in markdown" command as the canonical viewer? `sf plan show ` could render CONTEXT.md, ROADMAP.md, and VALIDATION.md on demand (lazy) without requiring the files to be on disk. This would enable a future lazy-only model. However, during the transition period, files must remain on disk for git diff and existing tool/agent consumers. The "view in markdown" command is worth adding as a convenience but does not replace disk presence in the MVP. ### Q3: Git diff UX — are DB state changes and markdown changes always committed atomically? This is the hardest open question. The current flow is: ``` DB write → disk write → autonomous loop continues → git commit (at unit boundary) ``` The git commit at unit boundary picks up all dirty files, which includes the regenerated markdown. So DB state changes and markdown changes should land in the same commit in the normal flow. **Risk:** if the session is interrupted between the DB write and the git commit, the DB has the new state but git does not. On the next run, the markdown is regenerated from DB (correct), but the git diff will show both the state change and the render as a single combined diff — which is correct and expected. **Unresolved:** for milestone-level artifacts (VALIDATION.md, CONTEXT.md, ROADMAP.md), the commit happens after `validate_milestone`/`plan_milestone`/`complete_milestone` returns. These tool calls run inside the agent turn, not at a unit boundary. The `postUnitPostVerification` flow (called after execute-task) would not catch a VALIDATION.md written during a `validate-milestone` unit. The commit hook for milestone tool calls needs explicit investigation — it may already be handled by the worktree merge path (`auto-worktree.js:syncMilestoneLevelFiles`), but this needs verification before Phase 4/5 rollout. --- ## Appendix A: File-to-DB Column Mapping | File | DB table | DB column(s) | |---|---|---| | `M*-VALIDATION.md` | `assessments` | `full_content`, `status` (= verdict), `scope=milestone-validation` | | `M*-ROADMAP.md` | `milestones` + `slices` | `title`, `status`, `sequence`, `risk`, `depends` | | `M*-ROADMAP.json` | Same as ROADMAP.md | Generated alongside ROADMAP.md | | `M*-CONTEXT.md` | `milestones` | `full_context_md` (additive), `vision`, `success_criteria`, `key_risks`, `proof_strategy`, `verification_*`, `definition_of_done`, `requirement_coverage`, `boundary_map_markdown` | | `M*-CONTEXT-DRAFT.md` | `milestones` | `context_draft_md` (additive) | | `M*-SUMMARY.md` | `milestones` | `full_summary_md` (additive), `status=complete`, `completed_at` | | `S*-SUMMARY.md` | `slices` | `full_summary_md` | | `S*-UAT.md` | `slices` | `full_uat_md`, `uat_verdict` | | `T*-SUMMARY.md` | `tasks` | `full_summary_md` | | `S*-PLAN.md` | `slices` | `goal`, `success_criteria`, `proof_level`, `integration_closure`, `planning_meeting_json` | | `T*-PLAN.md` | `tasks` | `full_plan_md`, `description`, `verify`, `inputs`, `expected_output` | --- ## Appendix B: Doctor Check Disposition | Doctor code | Status after full rollout | |---|---| | `db_projection_validation_drift` | **Removed** — file is always generated from DB; drift is impossible | | `db_projection_roadmap_drift` | **Removed** (Phase 3) — ROADMAP.json always generated alongside ROADMAP.md | | `db_missing_planning_rows_from_projection` | **Retained** — still needed during legacy migration; becomes a one-time import prompt | | `db_validation_content_missing` | **Added (MVP)** — file exists but DB has no assessment row | | `db_context_missing` | **Added (Phase 5)** — CONTEXT.md exists but `full_context_md` is empty | | `db_roadmap_not_imported` | **Added (Phase 3)** — ROADMAP.md has slices not in DB | | Stale render checks in `detectStaleRenders` | **Simplified** — reduced to recovery path only; normal-flow drift is impossible | --- ## Appendix C: Concrete Example — What Changes for M001-6377a4 The session failure was: 1. `M001-6377a4-VALIDATION.md` frontmatter said `verdict: needs-remediation`. 2. `assessments` table said `status: needs-attention`. 3. `doctor` detected drift and reported it. 4. Autonomous spent turns reconciling before re-running validation. After MVP (Step V1): - `complete-milestone.js` checks `getMilestoneValidationAssessment('M001-6377a4').status` instead of reading the file. If the DB says `needs-attention`, the milestone is blocked with `needs-attention` — correctly. - `workspace-index.js` populates `validationVerdict` from DB — no file read. - The stale VALIDATION.md is still on disk (and would be visible in git) but it is not consulted by any decision-making code. - The doctor check `db_projection_validation_drift` no longer fires because the read sites no longer compare file vs DB. A `sf doctor --fix` run would then re-render VALIDATION.md from the DB `assessments.full_content`, overwriting the stale `needs-remediation` frontmatter with the correct `needs-attention` verdict.