singularity-forge/docs/dev/proposals/db-first-planning-state.md

Proposal: DB-First Planning State — Markdown as Render Target

Status: Draft
Date: 2026-05-15
Author: Autonomous session + design review
Relates to: .sf/PRINCIPLES.md L1, docs/adr/0000-purpose-to-software-compiler.md

---

Executive Summary

SF's principle states "SQLite is the canonical structured store … treat .sf/sf.db as the first place for planning hierarchy." The implementation has not followed this: milestone artifact files (CONTEXT.md, ROADMAP.md, VALIDATION.md, SUMMARY.md, and their slice/task counterparts) are written and edited as primary sources of truth, with DB rows either reconciled after the fact or only partially kept in sync. The result is observable drift that costs the autonomous orchestrator agent turns and derails long runs.

This proposal describes how to complete the inversion: DB rows become the canonical source, markdown files become a deterministic render of that state, and the doctor/reconciler logic collapses to near-zero because there is nothing to reconcile.

---

1. Current State Survey

1.1 Artifact Inventory

The table below covers the artifact types found under .sf/milestones/M*/ (surveyed against M001-6377a4 as the reference case, confirmed by grep across the extension source).

Artifact	Example path	Written by	Read by	DB representation today	Authoritative today
M*-CONTEXT.md	M001-6377a4/M001-6377a4-CONTEXT.md	Bootstrap agent via write_file; record-promoter.js (migration path); migrate/writer.js	state-db.js buildRegistryAndFindActive (title extraction); reasoning-assist.js (injects into prompts); auto-start.js (presence check); auto-direct-dispatch.js	Partial: milestones.vision, vision_meeting_json, product_research_json, boundary_map_markdown, requirement_coverage hold structured fragments; the full narrative prose has no DB column	File — title is extracted from prose; structured fields exist but are not always populated
M*-CONTEXT-DRAFT.md	M001-6377a4/M001-6377a4-CONTEXT-DRAFT.md	Bootstrap agent via write_file; triage-resolution.js	state-db.js (presence check: draft → active milestone not yet planned); auto-start.js	None	File — only signal is presence/absence
M*-ROADMAP.md	M001-6377a4/M001-6377a4-ROADMAP.md	renderRoadmapFromDb in markdown-renderer.js; record-promoter.js; migrate/writer.js	state-db.js (fallback title when DB title blank); doctor-engine-checks.js (detectMissingPlanningRowsFromProjection); post-execution-checks.js (checkMilestoneIntegrity); commands-ship.js; commands-handlers.js	DB: milestones + slices tables hold all slice hierarchy, sequence, status, depends	Ambiguous — DB is authoritative for status/hierarchy; ROADMAP.md is a render target in theory but is also the fallback when DB is empty
M*-ROADMAP.json	M001-6377a4/M001-6377a4-ROADMAP.json	writeRoadmapJsonProjection (called from renderRoadmapFromDb)	doctor-engine-checks.js (projectionDriftIssues) — read-only drift check	DB: same milestones/slices tables	Projection — correctly labelled as "export/recovery only; DB remains authoritative" in doctor warning
M*-VALIDATION.md	M001-6377a4/M001-6377a4-VALIDATION.md	handleValidateMilestone in tools/validate-milestone.js (DB write first, then disk render)	state-db.js readMilestoneValidationVerdict (terminal check); doctor-engine-checks.js projectionDriftIssues; workspace-index.js (validationVerdict field); auto-prompts.js (injected as context); reflection.js; complete-milestone.js (re-reads verdict from file); auto-worktree.js	DB: assessments table (full_content, status/verdict, scope=milestone-validation)	Ambiguous — handleValidateMilestone writes DB first, then file; but complete-milestone.js re-reads the file, and workspace-index.js reads the file, not the DB
M*-SUMMARY.md	(milestone-level, e.g. M001-6377a4-SUMMARY.md)	handleCompleteMilestone in tools/complete-milestone.js	state-db.js buildRegistryAndFindActive (title fallback for complete milestones)	DB: milestone_evidence (narrative rendered) + milestone status/completed_at; no full_summary_md column on milestones table	File — absence of SUMMARY.md can block the completing-milestone phase even when DB says complete
S*-SUMMARY.md	slices/S01/S01-SUMMARY.md	Agent via write_file (autonomous solver), then renderSliceSummary backfills from DB slices.full_summary_md	summary-helpers.js (prior task context injection); doctor completeness check	DB: slices.full_summary_md TEXT column	Ambiguous — file written by agent first, DB column populated by setSliceSummaryMd; renderer then writes back from DB
S*-UAT.md	slices/S01/S01-UAT.md	Agent via write_file; renderSliceSummary (writes from slices.full_uat_md)	Doctor completeness check	DB: slices.full_uat_md TEXT column; slices.uat_verdict	Ambiguous — same dual-write pattern as SUMMARY.md
S*/tasks/T*-SUMMARY.md	slices/S01/tasks/T01-SUMMARY.md	Agent via write_file; renderTaskSummary (from tasks.full_summary_md)	summary-helpers.js prior-task context; doctor	DB: tasks.full_summary_md TEXT column	Ambiguous — task summary lives in DB but file is what the agent reads next turn
S*-PLAN.md / T*-PLAN.md	slices/S03/S03-PLAN.md	renderSlicePlanFromDb / renderTaskPlanFromDb in markdown-renderer.js	post-execution-checks.js; agent via read_file	DB: slices.goal, success_criteria, full_summary_md; tasks.full_plan_md, description, etc.	DB-first in theory — renderers exist; but slice PLAN is sometimes skipped if full_plan_md is empty
M*-CONTEXT-DRAFT.md (anchor)	anchors/plan-slice.json	plan tooling	auto-dispatch.js (planning anchor gate)	DB: none	File

1.2 Key Readers of Each Artifact

Who reads VALIDATION.md:

• state-db.js:readMilestoneValidationVerdict — terminal verdict check; blocks completing the milestone if not terminal
• workspace-index.js:buildWorkspaceIndex — populates validationVerdict on each milestone entry (used by TUI, context injection)
• auto-prompts.js:buildValidateMilestonePrompt — reads file as context block for re-validation rounds
• complete-milestone.js:handleCompleteMilestone — re-reads file verdict as guard before marking complete
• reflection.js:readValidationFiles — reads all milestone VALIDATION files for the reflection corpus
• doctor-engine-checks.js:projectionDriftIssues — compares file verdict vs DB assessments.status

Who reads CONTEXT.md:

• state-db.js:buildRegistryAndFindActive — title extraction fallback when milestones.title is blank
• reasoning-assist.js — injects context into prompt blocks
• auto-start.js — presence check (CONTEXT exists → bootstrap complete)
• auto-direct-dispatch.js — reads slice CONTEXT for dispatch context

Who reads ROADMAP.md:

• doctor-engine-checks.js:detectMissingPlanningRowsFromProjection — if ROADMAP.md has slices but DB has none, warns
• post-execution-checks.js:checkMilestoneIntegrity — reads ROADMAP.md to verify done slices have SUMMARY.md
• commands-ship.js — reads roadmap for ship-gate check
• state-db.js:534 — checks whether ROADMAP file exists as a proxy for "milestone has been planned"

Who reads SUMMARY.md (slice/task):

• summary-helpers.js — injected as prior-task context into execute-task prompts
• state-db.js:buildRegistryAndFindActive — reads milestone SUMMARY for title (complete milestone fallback)
• markdown-renderer.js:detectStaleRenders — checks presence of SUMMARY.md for complete tasks

1.3 Authoritative-Today Summary

Artifact            | Source of truth today    | Drift risk
--------------------|--------------------------|------------------
CONTEXT.md          | File (prose)             | High — title field in DB often blank
CONTEXT-DRAFT.md    | File (presence signal)   | Low (no content read)
ROADMAP.md          | Ambiguous — DB for status, file for completeness check | Medium
ROADMAP.json        | DB (projection)          | Low — doctor already catches drift
VALIDATION.md       | File (re-read by complete-milestone) | High — this session's critical failure
SUMMARY.md (ms)     | File (blocking)          | High — absence caused stuck phase
SUMMARY.md (slice)  | Ambiguous (dual-write)   | Medium
UAT.md (slice)      | Ambiguous (dual-write)   | Medium
SUMMARY.md (task)   | Ambiguous (dual-write)   | Medium
PLAN.md (slice/task)| DB-first via renderers   | Low (renderer exists)

---

2. Proposed Model

2.1 Canonical DB Representation (Reusing Existing Tables)

The existing schema already holds or can hold the authoritative state for every artifact type listed above. No new tables are required for the MVP; a few additive column additions are needed for the full rollout.

VALIDATION.md → assessments table

Already done (partially). handleValidateMilestone already writes assessments (full_content, status, scope=milestone-validation) before writing the file. The gap is that three consumers re-read the file instead of querying the DB.

No schema change required.

ROADMAP.md → milestones + slices tables

Already fully represented. Every slice's id, title, status, sequence, risk, depends, and done state are in the slices table. The renderRoadmapFromDb function in markdown-renderer.js already generates correct markdown from these rows.

No schema change required.

SUMMARY.md (milestone) → milestones + milestone_evidence tables

The milestone complete-milestone handler writes narrative to milestone_evidence but there is no milestones.full_summary_md column. Adding one mirrors the pattern used by slices.full_summary_md and tasks.full_summary_md.

Additive schema change required:

ALTER TABLE milestones ADD COLUMN full_summary_md TEXT NOT NULL DEFAULT '';

SUMMARY.md (slice) → slices.full_summary_md

Already exists. setSliceSummaryMd writes this column. renderSliceSummary reads it and writes to disk.

No schema change required.

UAT.md (slice) → slices.full_uat_md

Already exists. Same pattern as SUMMARY.md.

No schema change required.

CONTEXT.md → milestones table (structured fields) + new full_context_md column

The milestones table already holds structured fields extracted from CONTEXT.md (vision, success_criteria, key_risks, proof_strategy, verification_contract/integration/operational/uat, definition_of_done, requirement_coverage, boundary_map_markdown). What is missing is a column for the full human-authored narrative prose (the "Project Description", "Why This Milestone", "User-Visible Outcome" sections).

Adding full_context_md TEXT NOT NULL DEFAULT '' allows the complete context to be stored and rendered on demand.

Additive schema change required:

ALTER TABLE milestones ADD COLUMN full_context_md TEXT NOT NULL DEFAULT '';

CONTEXT-DRAFT.md → milestones.status + new context_draft_md column

The draft is a signal that planning is in progress. The presence check can be replaced by a column that stores the draft content; status queued with non-empty context_draft_md replaces the file presence check.

Additive schema change required:

ALTER TABLE milestones ADD COLUMN context_draft_md TEXT NOT NULL DEFAULT '';

2.2 Markdown as a Projection: Lazy vs Eager

Decision: Eager generation on write, with lazy fallback on read.

The render target must be written to disk at the same transaction boundary as the DB write, for two reasons:

1. Git atomicity: human reviewers see PR diffs. If the markdown file is absent or stale, the diff is hard to review. Committing DB state without a corresponding markdown update breaks review UX.
2. Existing readers that read the file: complete-milestone.js, workspace-index.js, reflection.js, auto-prompts.js all read the file path directly. Migrating all call sites at once is a large change; eager disk writes allow incremental migration.

Lazy generation (render only when cat or sf plan show is called) is appealing in the steady state but breaks the git-atomicity requirement and existing consumers during the transition period. A lazy-only model is the long-term ideal (addressed in Section 7), but not safe for the MVP.

The projection contract:

DB write → render markdown → atomic disk write → git commit
                ↑
         deterministic function;
         same DB state → same markdown content every time

The rendered file gets a comment header:

<!-- generated from .sf/sf.db — do not edit directly; use `sf plan` commands -->

For CONTEXT.md and CONTEXT-DRAFT.md, the "do not edit" header is softer because they contain human-authored prose that has no other editing path during the bootstrap phase. Editing UX is addressed in Section 3.

2.3 Migration: Existing Markdown → DB Import

For each artifact type, the migration follows this pattern:

1. Read the existing markdown file.
2. Parse the structured content using existing parsers (parseRoadmap, extractVerdict, parseSummary, frontmatter parser).
3. Upsert the parsed content into the appropriate DB column(s) using existing functions (e.g., upsertMilestonePlanning, setSliceSummaryMd, insertAssessment).
4. Re-render the file from DB (which adds the <!-- generated --> header and normalizes formatting).
5. If the re-rendered content differs from the original, commit as a single atomic operation.

One-time migration function location: src/resources/extensions/sf/migrate/db-first-import.js. It is called once by doctor --fix when the db_markdown_not_imported doctor code fires; never called on every startup.

The migration is idempotent: if assessments.full_content already matches the file's content hash, it is a no-op.

---

3. Editing UX

3.1 Options

(a) Edit markdown → parse back into DB (round-trip-safe)

Requires a robust bidirectional parser for every artifact type. CONTEXT.md is free-form prose; there is no safe round-trip parser. Any ambiguous or model-generated prose in CONTEXT.md would be lost, truncated, or corrupted on re-import. High implementation cost; fragile.

(b) Markdown is read-only generated; edits go through sf plan commands or sf headless db ...

Clean separation. Agents and operators must use structured commands to mutate state. The markdown file is authoritative only for reading/reviewing.

(c) Hybrid: drafts are markdown, promotion to canonical writes DB and regenerates markdown

CONTEXT-DRAFT.md remains agent-editable (it is the active working surface during bootstrap). Once plan_milestone is called, CONTEXT.md is generated from DB and becomes read-only.

Decision: Option (c) — hybrid.

Reasoning:

• CONTEXT-DRAFT.md and slice/task PLAN.md are the primary LLM write surfaces during active work. Making them DB-only would require adding LLM tool calls for every sentence of the planning meeting. The bootstrap and planning flows are already LLM-driven with free-form file writes.
• VALIDATION.md, ROADMAP.md, SUMMARY.md, and UAT.md are outputs of structured operations (validate_milestone, plan_milestone, complete_slice, complete_task). These have clean tool-call boundaries and can safely become DB-first with no editing path for the file itself.
• The hybrid lets us complete the MVP without restructuring the bootstrap/planning LLM flow, while still eliminating the drift that caused the session failures.

Editing rules under the hybrid model:

Artifact	Editing path	File state
CONTEXT-DRAFT.md	Agent writes via write_file during bootstrap; content stored in milestones.context_draft_md on save	Editable (agent/human)
CONTEXT.md	plan_milestone call generates from DB fields + draft content; file becomes read-only	<!-- generated --> header
ROADMAP.md	plan_milestone / sf plan update-slice commands; file is regenerated	<!-- generated --> header
ROADMAP.json	Regenerated alongside ROADMAP.md	// generated comment
VALIDATION.md	validate_milestone tool call; file regenerated from DB	<!-- generated --> header
SUMMARY.md (slice/task)	complete_slice / complete_task tool calls; field written to DB, file generated	<!-- generated --> header
UAT.md	complete_slice tool call (uat_md param); field written to DB	<!-- generated --> header
SUMMARY.md (milestone)	complete_milestone tool call; milestone full_summary_md column written, file generated	<!-- generated --> header

For operators who need to edit a generated file (e.g., to correct a validation rationale), the path is:

1. sf headless db -- "UPDATE assessments SET full_content = '...' WHERE milestone_id = 'M001'" — or via sf plan validate --update-rationale.
2. The update triggers a projection re-render (file is rewritten from DB).

---

4. Stale-Detection and Reconciliation

4.1 Current Reconciliation Cost

Today, doctor-engine-checks.js:projectionDriftIssues performs two checks:

1. ROADMAP.json drift: compares DB slice order/status against JSON file. Fires db_projection_roadmap_drift warning.
2. VALIDATION.md verdict drift: compares DB assessments.status against file frontmatter verdict. Fires db_projection_validation_drift warning. This is the check that fired during the M001-6377a4 session ("verdict in markdown said needs-remediation but DB said needs-attention").

Additionally, detectStaleRenders in markdown-renderer.js checks checkbox states and missing SUMMARY.md files. workspace-index.js reads VALIDATION files independently of the DB. state-db.js reads CONTEXT.md for title extraction when the DB title is blank.

Each of these is a potential source of drift. The autonomous orchestrator spent turns reasoning about the discrepancy before acting on it.

4.2 After DB-First Migration: Doctor Changes

For VALIDATION.md: The db_projection_validation_drift check is eliminated entirely. The file is always regenerated from DB by handleValidateMilestone. The doctor check becomes:

if (validationPath exists AND full_content in assessments is empty) {
  issue: db_validation_content_missing — file exists but DB has no stored content;
         run `sf doctor --fix` to import the file into DB.
  severity: info, fixable: true
}

This inverts the check: instead of asking "does the file match the DB?", it asks "is the DB populated for this file?". Once the one-time import runs, the check never fires again.

For ROADMAP.json: The db_projection_roadmap_drift check remains as a sanity assertion during the transition period. After full rollout, it is removed: the JSON file is always regenerated alongside ROADMAP.md by renderRoadmapFromDb.

For SUMMARY.md / UAT.md missing files: detectStaleRenders currently detects when a complete task/slice has content in DB but no file on disk. After DB-first, the render is triggered eagerly by complete_task/complete_slice, so the file is never missing. The check becomes a recovery path only (fired when the disk write failed mid-way).

For CONTEXT.md title fallback: state-db.js:buildRegistryAndFindActive falls back to reading CONTEXT.md when milestones.title is blank. After DB-first, the full_context_md column is populated on every plan_milestone call, and milestones.title is always written alongside it. The file-system fallback path is removed from buildRegistryAndFindActive.

For CONTEXT.md presence check in auto-start.js: The check if (!contextFile && draftFile) activeMilestoneHasDraft = true is replaced by checking milestones.context_draft_md IS NOT NULL AND context_draft_md != ''.

Net result: the doctor has nothing to reconcile for these artifact types. The autonomous orchestrator spends zero turns on drift detection. The estimated ~20% reconciliation overhead in early autonomous runs goes to near-zero.

---

5. Migration Plan

5.1 MVP: *-VALIDATION.md

Why first:

• Smallest file (frontmatter + ~30 lines); parser (extractVerdict) already exists and is battle-tested.
• The doctor already detects drift (db_projection_validation_drift), providing a clear before/after signal.
• handleValidateMilestone already writes DB before disk — the pattern is already correct at the write site. The remaining work is fixing the three read sites.
• The session failure was specifically caused by stale VALIDATION.md verdict. This is the highest-pain point.

Step-by-step for VALIDATION.md MVP:

Step V1: Fix read sites to prefer DB.

Files to change:

• complete-milestone.js:handleCompleteMilestone — line 136-142: replace extractVerdict(readFileSync(validationPath)) with getMilestoneValidationAssessment(milestoneId)?.status.
• workspace-index.js:buildWorkspaceIndex — replace file-read loop with getMilestoneValidationAssessment(milestone.id)?.status.
• state-db.js:readMilestoneValidationVerdict — replace file-read with getMilestoneValidationAssessment(milestoneId)?.status; keep file-read as fallback only when DB row is absent (for repos that have not yet run the migration).

Rollback: trivially revert the three call-site changes. No schema change, no data loss.

Readers that do NOT switch to DB in this step:

• auto-prompts.js:buildValidateMilestonePrompt — still reads the file as a human-readable context block (acceptable; the file is generated from DB so it will be correct).
• reflection.js:readValidationFiles — still reads files for reflection corpus (acceptable; same reason).

Tests:

• Add a test in src/resources/extensions/sf/tests/ that calls handleValidateMilestone, then checks that getMilestoneValidationAssessment returns the verdict — and that the same verdict is returned by readMilestoneValidationVerdict without any file on disk.
• Extend doctor-engine-checks tests to confirm that db_projection_validation_drift no longer fires for a freshly validated milestone.

Step V2: Add <!-- generated --> header to VALIDATION.md output.

In renderValidationMarkdown inside tools/validate-milestone.js, prepend:

<!-- generated from .sf/sf.db — do not edit directly; use validate_milestone tool -->

Update verdict-parser.js:extractVerdict to skip this comment line when parsing verdicts from files.

Tests: confirm that extractVerdict still works on files with the header; confirm that the doctor check does not false-positive on the header.

Step V3: One-time import doctor fix.

Add doctor code db_validation_content_missing: fires when a VALIDATION.md file exists for a milestone but assessments has no row for scope=milestone-validation for that milestone. Doctor --fix calls insertAssessment with the file content.

Tests: set up a milestone with a VALIDATION.md but no DB row; confirm doctor fires and fix imports it.

Step V4: Remove db_projection_validation_drift doctor check.

Once V1–V3 are in and the migration has run, the drift check is provably unreachable (the file is always generated from DB). Remove it to reduce noise.

Rollback at any step: each step is independently revertible. The DB assessments rows are not deleted during any step, so reverting the read-site changes restores the file-read behavior.

---

5.2 Incremental Rollout to CONTEXT.md

Dependencies: CONTEXT.md is not currently in the DB at the full-prose level. The full_context_md schema addition must land first.

Sequence:

1. Add milestones.full_context_md TEXT NOT NULL DEFAULT '' migration (schema version bump in sf-db-schema.js).
2. Update plan_milestone tool handler to write the full context prose to full_context_md in addition to the existing structured fields.
3. Update state-db.js:buildRegistryAndFindActive title extraction to read from DB title first (already done for non-empty titles); add fallback to extract title from full_context_md before falling back to file.
4. Update triage-resolution.js CONTEXT-DRAFT.md seed to also write milestones.context_draft_md.
5. Add renderContextFromDb function to markdown-renderer.js that generates CONTEXT.md from full_context_md (or structured fields if full_context_md is empty).
6. Update plan_milestone to call renderContextFromDb after writing DB, replacing the direct write_file instruction in the LLM prompt for the CONTEXT.md write.
7. Doctor check: db_context_missing fires when CONTEXT.md exists but full_context_md is empty; --fix imports file content.
8. CONTEXT-DRAFT.md: auto-start.js presence check replaced by milestones.context_draft_md IS NOT NULL.

Tests: planning flow integration test that verifies plan_milestone writes full_context_md to DB and regenerates CONTEXT.md with correct content and header.

---

5.3 Incremental Rollout to ROADMAP.md

ROADMAP.md is already partially DB-first: renderRoadmapFromDb exists and is called by the slice/task completion path. The remaining work is:

1. Ensure plan_milestone always calls renderRoadmapFromDb instead of (or in addition to) writing ROADMAP.md via direct file instruction.
2. Replace state-db.js:534 file-existence proxy (resolveMilestoneFile(basePath, activeMilestone.id, 'ROADMAP') !== null) with a DB check (getMilestoneSlices(milestoneId).length > 0).
3. Replace post-execution-checks.js:checkMilestoneIntegrity ROADMAP read with a direct DB query for done slices, then check SUMMARY.md presence separately.
4. Add <!-- generated --> header; update parseRoadmap to skip the header line.
5. Doctor check: db_roadmap_not_imported fires when ROADMAP.md has slices not in DB; --fix calls migrateHierarchyToDb (already exists in md-importer.js).

---

5.4 Incremental Rollout to SUMMARY.md (Milestone)

1. Add milestones.full_summary_md TEXT NOT NULL DEFAULT '' schema migration.
2. Update handleCompleteMilestone to write full_summary_md to the milestones table.
3. Add renderMilestoneSummaryFromDb to markdown-renderer.js.
4. Call the renderer from handleCompleteMilestone (replacing the direct saveFile call or running alongside it during transition).
5. Update state-db.js:buildRegistryAndFindActive title fallback for complete milestones to read from milestones.full_summary_md (or milestones.title) before reading the file.
6. This eliminates the "SUMMARY.md missing → completing-milestone phase stuck" failure mode: the DB row is written first, and the render is a deterministic step.

---

5.5 Rollout Order Summary

Phase	Artifact	Required schema change	Estimated effort
MVP	VALIDATION.md	None	Small (3 read-site changes + tests)
2	SUMMARY.md (slice, task)	None (columns exist)	Small (add header, fix stale-render fallback)
3	ROADMAP.md	None	Medium (dispatch-side guard change + test)
4	SUMMARY.md (milestone)	milestones.full_summary_md	Medium
5	CONTEXT.md	milestones.full_context_md	Large (LLM flow change)
6	CONTEXT-DRAFT.md	milestones.context_draft_md	Small (presence check replacement)

---

6. Non-Goals

• Does NOT replace .sf/journal/*.jsonl (append-only event log; complementary to DB; no change).
• Does NOT replace ADRs / proposals in docs/ (human-authored; promoted-only; not generated). See PRINCIPLES.md L4: "promote only reviewed plans, specs, and ADRs to docs/".
• Does NOT abolish markdown. Markdown files under .sf/milestones/ remain on disk and are committed to git. They are readable by humans and LLMs. The only change is that they are generated artifacts rather than primary state. The git history of these files remains meaningful as a timeline of validation and completion events.
• Does NOT change the slice PLAN.md or task PLAN.md write flow. These are already DB-first (via renderSlicePlanFromDb / renderTaskPlanFromDb). The full_plan_md column exists on tasks. No migration needed.
• Does NOT change .sf/runtime/ files (solver checkpoint, unit runtime record). These are ephemeral operational state, not planning state.
• Does NOT change docs/records/ workflow. The record-promoter.js promotes human-authored records to milestone shells; those shells still use CONTEXT-DRAFT.md as the initial planning surface.
• Does NOT gate on a single DB technology. The proposal assumes SQLite via sf-db.js. No change to the database engine.

---

7. Open Questions

Q1: How are DB-stored rich-text fields exported to markdown (rendering rules)?

The renderValidationMarkdown and renderRoadmapMarkdown functions in markdown-renderer.js and validate-milestone.js already implement this for their respective artifact types. For CONTEXT.md, the rich-text is free-form prose stored verbatim in full_context_md; the renderer wraps it in the standard header/metadata block. No special encoding or schema migration is needed for plain markdown prose fields.

For structured list fields (e.g., milestones.success_criteria stored as JSON array), the existing renderListEntry helper in markdown-renderer.js handles the serialization. The schema migration path (adding a column) is gated on the existing columnExists helper and the versioned migration table in sf-db-schema.js.

Unresolved: What happens when the free-form prose in full_context_md contains markdown that conflicts with the generated header/section structure? The current normalizeMarkdownBlockSpacing helper handles some cases, but adversarial LLM-authored content with duplicate H1 headings or frontmatter blocks could corrupt the render. A content sanitizer for known problematic patterns should be added to the render pipeline before Phase 5 (CONTEXT.md).

Q2: Should there be a "view in markdown" command as the canonical viewer?

sf plan show <milestone-id> could render CONTEXT.md, ROADMAP.md, and VALIDATION.md on demand (lazy) without requiring the files to be on disk. This would enable a future lazy-only model. However, during the transition period, files must remain on disk for git diff and existing tool/agent consumers. The "view in markdown" command is worth adding as a convenience but does not replace disk presence in the MVP.

Q3: Git diff UX — are DB state changes and markdown changes always committed atomically?

This is the hardest open question. The current flow is:

DB write → disk write → autonomous loop continues → git commit (at unit boundary)

The git commit at unit boundary picks up all dirty files, which includes the regenerated markdown. So DB state changes and markdown changes should land in the same commit in the normal flow.

Risk: if the session is interrupted between the DB write and the git commit, the DB has the new state but git does not. On the next run, the markdown is regenerated from DB (correct), but the git diff will show both the state change and the render as a single combined diff — which is correct and expected.

Unresolved: for milestone-level artifacts (VALIDATION.md, CONTEXT.md, ROADMAP.md), the commit happens after validate_milestone/plan_milestone/complete_milestone returns. These tool calls run inside the agent turn, not at a unit boundary. The postUnitPostVerification flow (called after execute-task) would not catch a VALIDATION.md written during a validate-milestone unit. The commit hook for milestone tool calls needs explicit investigation — it may already be handled by the worktree merge path (auto-worktree.js:syncMilestoneLevelFiles), but this needs verification before Phase 4/5 rollout.

---

Appendix A: File-to-DB Column Mapping

File	DB table	DB column(s)
M*-VALIDATION.md	assessments	full_content, status (= verdict), scope=milestone-validation
M*-ROADMAP.md	milestones + slices	title, status, sequence, risk, depends
M*-ROADMAP.json	Same as ROADMAP.md	Generated alongside ROADMAP.md
M*-CONTEXT.md	milestones	full_context_md (additive), vision, success_criteria, key_risks, proof_strategy, verification_*, definition_of_done, requirement_coverage, boundary_map_markdown
M*-CONTEXT-DRAFT.md	milestones	context_draft_md (additive)
M*-SUMMARY.md	milestones	full_summary_md (additive), status=complete, completed_at
S*-SUMMARY.md	slices	full_summary_md
S*-UAT.md	slices	full_uat_md, uat_verdict
T*-SUMMARY.md	tasks	full_summary_md
S*-PLAN.md	slices	goal, success_criteria, proof_level, integration_closure, planning_meeting_json
T*-PLAN.md	tasks	full_plan_md, description, verify, inputs, expected_output

---

Appendix B: Doctor Check Disposition

Doctor code	Status after full rollout
db_projection_validation_drift	Removed — file is always generated from DB; drift is impossible
db_projection_roadmap_drift	Removed (Phase 3) — ROADMAP.json always generated alongside ROADMAP.md
db_missing_planning_rows_from_projection	Retained — still needed during legacy migration; becomes a one-time import prompt
db_validation_content_missing	Added (MVP) — file exists but DB has no assessment row
db_context_missing	Added (Phase 5) — CONTEXT.md exists but full_context_md is empty
db_roadmap_not_imported	Added (Phase 3) — ROADMAP.md has slices not in DB
Stale render checks in detectStaleRenders	Simplified — reduced to recovery path only; normal-flow drift is impossible

---

Appendix C: Concrete Example — What Changes for M001-6377a4

The session failure was:

1. M001-6377a4-VALIDATION.md frontmatter said verdict: needs-remediation.
2. assessments table said status: needs-attention.
3. doctor detected drift and reported it.
4. Autonomous spent turns reconciling before re-running validation.

After MVP (Step V1):

• complete-milestone.js checks getMilestoneValidationAssessment('M001-6377a4').status instead of reading the file. If the DB says needs-attention, the milestone is blocked with needs-attention — correctly.
• workspace-index.js populates validationVerdict from DB — no file read.
• The stale VALIDATION.md is still on disk (and would be visible in git) but it is not consulted by any decision-making code.
• The doctor check db_projection_validation_drift no longer fires because the read sites no longer compare file vs DB.

A sf doctor --fix run would then re-render VALIDATION.md from the DB assessments.full_content, overwriting the stale needs-remediation frontmatter with the correct needs-attention verdict.