Comprehensive guide for migrating from JSON to node:sqlite when Node 24 is available:
- Schema design (model_outcomes + model_stats tables)
- Phase-by-phase refactoring approach
- Data migration from JSON with backward compatibility
- Testing strategy with new SQLite-specific tests
- Future opportunities: dashboards, trend analysis, A/B testing, federated learning
This doc serves as a roadmap for ~2 days of work when Node 24 becomes standard.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Integration of 3 quick wins into existing UOK infrastructure:
1. Model Learning (Quick Win #2) → metrics.js
- Record outcomes to model-learner for per-task-type performance tracking
- Hook: recordUnitOutcome() now calls ModelLearner.recordOutcome()
- Fire-and-forget: never blocks outcome recording on learning failure
- Enables adaptive model routing decisions in downstream gates
2. Self-Report Fixing (Quick Win #1) → triage-self-feedback.js
- Auto-fix high-confidence reports (>0.85) in applyTriageReport()
- Hook: After triage and requirement promotion, apply auto-fixes
- Fire-and-forget: never blocks report application on fix failure
- Returns reportsAutoFixed count for triage metrics
3. Knowledge Injection (Quick Win #3) → already integrated in auto-prompts.js
- Already active in execute-task prompt template
- Semantic matching with graceful degradation
All integration points:
- Fire-and-forget: learning/fixing failures never block dispatch
- UOK-native: use existing outcome recording, db, gates
- Backward compatible: applyTriageReport now async, but callers handle it
- No new dependencies: all modules already in codebase
Testing: 2934 tests pass (no regressions from integration)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Updates 23c5de38b (which flagged the table as storage-only) to reflect
that 55b14c3f7 wired the ranker consumer (graph-boost in
getRelevantMemoriesRanked) and b9bff3762 wired the writer
(co-extraction linkage in applyMemoryActions). The graph-aware
pipeline is now end-to-end live, with named relation types,
auto-linking confidence (0.5), intra-pool boost, and damping (0.4).
Honest description for contributors reading top-down.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The architecture.md entry implied memory-relations.ts contributes to
ranking ("knowledge-graph edges between memories"). The read consumer
doesn't exist yet — getRelevantMemoriesRanked uses cosine + static
score, not graph traversal. Relations are written via /sf memory
import / createMemoryRelation but never read for ranking.
Updated the description so a contributor reading this file knows the
graph-traversal pipeline is the next logical extension, not something
that currently runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The repo's architecture file listed only `memory-extractor.ts` and
`memory-store.ts` — the rest of the memory subsystem
(`memory-embeddings.ts`, `memory-embeddings-llm-gateway.ts`,
`memory-relations.ts`, `memory-source-store.ts`) had no entry, so a
new contributor reading the file would miss them entirely.
Added one-line descriptions for each, including the gateway adapter's
opt-in env-var contract (`SF_LLM_GATEWAY_KEY`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
prompts/parallel-research-slices.md step 3 told the dispatcher to verify
research at `.sf/{{mid}}/`, but slice research files actually live at
`.sf/milestones/{{mid}}/slices/<sliceId>/<sliceId>-RESEARCH.md`. Step 3
verification could only ever fail.
prompts/validate-milestone.md sent the three milestone-validation reviewer
agents to wrong paths:
- parentTrace pointed at `.sf/{{milestoneId}}/S0X-SUMMARY.md` (slice
summaries actually live at `.sf/milestones/{{milestoneId}}/slices/S0X/`)
- Reviewer A read `.sf/{{milestoneId}}/REQUIREMENTS.md` (the file is at
project-level `.sf/REQUIREMENTS.md`)
- Reviewer A scanned `.sf/{{milestoneId}}/` for slice SUMMARYs (wrong dir)
- Reviewer C read `.sf/{{milestoneId}}/CONTEXT.md` (actual file is
`.sf/milestones/{{milestoneId}}/{{milestoneId}}-CONTEXT.md`)
Reviewers would either return false MISSING / FAIL verdicts or have to
re-discover the layout.
docs/dev/ADR-{008,009}-IMPLEMENTATION-PLAN.md "Related ADR" links pointed
to absolute paths inside a contributor's old Mac (`/Users/jeremymcspadden/
Github/sf-2/...`). Replaced with sibling-file relative paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After fixing forensics.md and error-classifier.ts last fire, swept the
rest of the tree for the same class of stale reference:
- scripts/validate-pack.js: criticalPackages list used \`@sf\` and
\`@sf-build\` scopes — neither exists in node_modules; this is in CI
(.github/workflows/ci.yml) + prepublishOnly, so the validation step
was failing to find anything. Now \`@singularity-forge/pi-coding-agent\`
and \`@singularity-forge/rpc-client\` (the actual scope).
- src/resources/skills/github-workflows/references/gh/SKILL.md: same
GraphQL bug as forensics.md — owner:"sf-build" name:"sf-2" — and
three \`gh project\` commands using owner sf-build. The gh issue
create command above already used singularity-forge/sf-run, so the
follow-up calls always failed. Also retitled "sf-2 Backlog" to
"sf-run Backlog".
- src/resources/extensions/sf/bootstrap/system-context.ts: deprecation
warning linked to https://github.com/sf-build/SF/issues/1492.
- packages/mcp-server/README.md, packages/rpc-client/README.md: 9 refs
to \`@sf-build/...\` for installable package names — would mislead
anyone copy-pasting into npm install.
- docs/user-docs/troubleshooting.md (+ zh-CN): GitHub Issues link
pointed at github.com/sf-build/SF/issues.
- docs/user-docs/getting-started.md (+ zh-CN): clone URL was correct
but the next \`cd\` was \`cd sf-2/docker\` — won't exist after a
fresh clone of sf-run.
- docs/dev/ci-cd-pipeline.md: GHCR org was \`sf-build\`.
Code comments containing "sf-2" / "sf-build" in non-active places
(parsers.ts banner, error message URLs in tests, dev-doc absolute
paths from a contributor's Mac) left alone — they're informational
and not addressed by users or runtime.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- engines.node: >=24.15.0 across all 23 package.json (root + 8
workspace + studio + web + pkg + vscode-extension + 11 SF
extension manifests)
- CI workflows pinned to node-version: '24.15' (16 sites)
- Dockerfile -> node:24.15-slim
- .nvmrc / .node-version -> 24.15.0
- Refactored worktree-cli.ts and headless-query.ts to use
import.meta.filename instead of fileURLToPath(import.meta.url)
- exec.ts simplified with AbortSignal.any + spawn signal/killSignal
- Picks up Crush's biome.json + AGENTS.md doc cleanup in same pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tail-end of the PDD v2 work (Assumptions field + safety/liveness split +
machine-executable Evidence). Three documents that still referenced v1's
4-field Purpose Gate are updated to the full 8-field PDD packet:
- docs/SPEC_FIRST_TDD.md — Purpose Gate now lists all 8 fields with the
Assumptions and Failure-boundary additions inline.
- skills/requesting-code-review — replaces "Purpose & Consumer" section with
"PDD packet (all 8 fields)" restated verbatim from .sf/active/{unit-id}/pdd.md.
Falsifier and Scope-defence sections clarified vs Failure-boundary and
Non-goals to remove overlap.
- skills/receiving-code-review — Purpose Gate criterion updated to demand
the full PDD packet with machine-executable Evidence, not just
Purpose/Consumer/Value-at-risk.
PDD packet (inline):
- Purpose: every artefact that references "Purpose Gate" agrees on the same
8-field definition; reviewers and reviewees read the same packet.
- Consumer: spec-first-tdd, requesting-code-review, receiving-code-review.
- Contract: all three documents list the same 8 fields with the same
Assumptions / safety+liveness / machine-executable-Evidence wording.
- Evidence: grep confirms PDD packet references in all three; typecheck:extensions exits 0.
- Non-goals: no edits to the PDD skill itself (already v2); no edits to other
skills referencing v1 Purpose Gate beyond these three (they don't exist).
- Invariants: existing review-loop sections preserved; only Purpose-Gate-
related sections rewritten.
- Assumptions: PDD v2 SKILL.md is the canonical source of field definitions;
these three documents are projections of it.
Performance fix from audit:
- bootstrap/system-context.ts: cachedReadFile() with mtime-keyed in-process
cache for KNOWLEDGE.md (global + project) and ARCHITECTURE.md. Eliminates
3-4 sync readFileSync calls per agent turn on the common case where these
files haven't changed. Live edits still picked up via mtime invalidation.
Docstring sweep on the notification + detection cluster:
- headless-events.ts: 17 JSDoc blocks (exit codes + every classification fn)
- notification-store.ts, notification-overlay.ts, notification-widget.ts,
notifications.ts: ~17 blocks
- detection.ts, codebase-generator.ts: ~5 blocks
Typecheck clean. 3/3 perf tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Generalizes the preferences-template-upgrade pattern to all scaffold-managed
documents with three states (pending/editing/completed), HTML-comment markers
on Markdown files, frontmatter on PREFERENCES.md, and a content-hash archive
for migrating legacy projects.
Operation is automatic-first, not command-driven:
- Synchronous on every SF startup (cheap path: missing + upgradable + legacy)
- Asynchronous after milestone completion: scaffold-keeper subagent runs the
existing records-keeper skill, treating code as the source of truth and
re-deriving doc content from source when drift is detected
- Surfaces results via the structured-notification model (kind:approval_request)
only when human review is warranted; silent runs produce no notification
- Manual /sf scaffold sync exists as an escape hatch for dry-run + forced
refresh, not as the primary interface
Five implementation phases (A-E), each independently shippable. Phase A
unlocks the architectural property; Phase D is what makes records-keeper
autonomous for code-derived docs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 1 — close SF-side polish gaps:
- codebase-generator: distinguish uv/poetry/pdm in Python stack-signals;
surface configured tooling (ruff/mypy/pyright) when config files exist
- doctor-environment: new checkPythonEnvironment — detects uv/poetry/pdm
via lockfile, verifies binary on PATH, warns with install hint when missing
- doctor-environment: new checkSiftAvailable — recommends sift install for
repos > 5000 source files when not on PATH
- tech-debt-tracker: documented future memory-as-sub-extension extraction
(defer until real backend-swap requirement)
Phase 2 — internal wire architecture:
- ADR-020: singularity-grpc as shared schema repo; gRPC + typed clients
for first-party services; MCP façade only at external-tool boundary
- ADR-019: trimmed MCP scope section to a 3-line summary linking to ADR-020
to avoid the wire-format table living in two places
- design-docs/index.md: ADR-020 added to ADR table
These changes make SF stronger for autonomous work on Python repos
(particularly ace-coder) and capture the internal wire architecture
decision as a durable ADR before any singularity-grpc code lands.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ADR-019 framing corrections:
- SF is single-machine, single-user, single-repo by design — character, not
limitation. Stays a standalone app permanently; does not get absorbed into ACE.
- Phase 6 reframed: "pattern transfer" not "orchestration convergence." ACE
ports patterns from SF, both apps remain independent.
- Phase 2 reframed: SF stays local. Federation is an ACE concern; SF doesn't
wire memory-store remote-mode against singularity-memory.
Detection strengthened for Python (priority for ace-coder work):
- Detect uv / poetry / pdm and prefix verification commands accordingly
- Emit ruff check when configured (file or [tool.ruff] in pyproject.toml)
- Emit mypy / pyright when configured — skip when no config to avoid false fails
- pyprojectHasTool helper for [tool.<name>] section detection
Detection strengthened for Rust:
- cargo fmt --check (fastest, catches style first)
- cargo check (type-only, faster than test)
- cargo clippy -- -D warnings (warnings as errors)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Internal services (SF↔memory, ACE↔memory, SF↔ACE) talk via typed direct
clients generated from the Go/TS APIs — HTTP/gRPC for memory, existing
JSON-RPC stdio for SF↔ACE. MCP is reserved for external LLM-driven coding
tools (Claude Code, Cursor) that don't share our build system; it is a
scaffold for the period when external coders help build the platform and
shrinks as the system becomes self-hosting.
Adds an explicit "MCP scope" table so the rule is stated once. Updates the
three-layer architecture diagram, Phase 2, and Phase 6 to remove the
inaccurate "all consumers over MCP" framing.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Captures the SF↔ACE incremental convergence strategy: workspace VMs
(Firecracker) as the unified execution isolation primitive, the three-layer
architecture (orchestration/knowledge/execution), the 6-phase convergence
path, and ADR-014 Phase 4 cancellation (persistent-agent runtime reassigned
to ACE). Cross-references the matching ACE document at
docs/architecture/sf-ace-convergence.md.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add harness/ directory to SF repo (specs/, evals/, graders/ with AGENTS.md)
and seed harness/specs/bootstrap.md (agent-legibility verification)
- Extend agentic-docs-scaffold.ts: new repos get harness/ + ADR-TEMPLATE.md
and just adr / just spec / just harness-spec recipes via justfile
- Sync SF_RUNTIME_PATTERNS (gitignore.ts canonical) → git-service.ts and
worktree-manager.ts: add audit/, exec/, model-benchmarks/, reports/,
notifications.jsonl, routing-history.json, self-feedback.jsonl, repo-meta.json,
and milestone continue-marker patterns
- Inject ARCHITECTURE.md into system prompt via loadArchitectureBlock() in
system-context.ts (capped at 8 000 chars, after KNOWLEDGE block)
- Write real ARCHITECTURE.md for this repo (system map, .sf/ layout, key flows)
- Add ADR-TEMPLATE.md to docs/design-docs/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gitignore (core change):
- Remove stale blanket .sf/ entries from .gitignore (migrated to
.git/info/exclude on 2026-04-29, never cleaned up)
- gitignore.ts: split SF_RUNTIME_EXCLUSION_PATTERNS into two modes —
SF_SYMLINK_EXCLUSION_PATTERNS (blanket .sf for symlink repos where
git cannot traverse the symlink) and SF_RUNTIME_EXCLUSION_PATTERNS
(granular runtime-only patterns for directory repos, enabling
.sf/milestones/ and other durable planning artifacts to be tracked)
- ensureGitInfoExclude() now detects symlink vs directory and writes
the correct patterns, handling transitions between modes cleanly
- ADR-001 status: Proposed → Accepted
Docs:
- Fill 11 placeholder scaffold docs with real SF-specific content:
PLANS, DESIGN, PRODUCT_SENSE, QUALITY_SCORE, RELIABILITY, SECURITY,
design-docs/index.md, exec-plans/active, exec-plans/completed,
exec-plans/tech-debt-tracker, records/index
- Add records note: docs/records/2026-05-01-repo-vcs-and-notifications.md
- ADR-008 status: Accepted → Proposed (deferred — not applicable to
current usage model where Claude Code assists externally, not as a
Pi provider inside SF's dispatch loop)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add repository-vcs-context.ts to detect and inject VCS context (Git/Jujutsu)
into the agent system prompt; wire in repo-vcs bundled skill trigger
- Add src/resources/skills/repo-vcs/ skill for commit, push, and safe-push workflows
- Add JSDoc Purpose/Consumer annotations to app-paths, bundled-extension-paths,
errors, extension-discovery, extension-registry, headless-types, headless, and traces
- Add justfile and just to flake.nix devShell
- Fill out new-user-onboarding.md spec (Draft) and core-beliefs.md (Status: Accepted)
- Add notification-event-model.md design doc and notification-source-hygiene.md spec
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
