From e679478d1bead9a6821876ae502b6f4e42e55a60 Mon Sep 17 00:00:00 2001 From: Mikael Hugo Date: Mon, 11 May 2026 19:24:23 +0200 Subject: [PATCH] feat(wiki): wire .sf/wiki/ as tracked context source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - auto-bootstrap-context.js: scan .sf/wiki/*.md in collectAutoBootstrapFiles so wiki pages load as priority context in headless autonomous bootstrap - headless-context.ts: same fix for the TS bootstrap path - system-context.js: loadWikiBlock already existed and was wired into fullSystem; add .sf/wiki/ to Tier 1 escalation policy lookup sources - system.md: add wiki/ to .sf/ directory structure; add Conventions entry explaining wiki is tracked in git (hand edits persist) and injected automatically when present - git-runtime-patterns.js: do NOT gitignore .sf/wiki/ — wiki pages are tracked like DECISIONS.md so hand edits survive commits and clones - .sf/wiki/: seed index.md, architecture.md, workflows.md for this repo Wiki filenames follow sf-wiki SKILL.md convention: lowercase (index.md, architecture.md, workflows.md, subsystems.md, glossary.md). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .sf/wiki/architecture.md | 67 ++++++++++++++++ .sf/wiki/index.md | 53 +++++++++++++ .sf/wiki/workflows.md | 76 +++++++++++++++++++ src/headless-context.ts | 24 ++++++ .../extensions/sf/auto-bootstrap-context.js | 22 ++++++ .../extensions/sf/bootstrap/system-context.js | 53 ++++++++++++- src/resources/extensions/sf/prompts/system.md | 7 ++ 7 files changed, 299 insertions(+), 3 deletions(-) create mode 100644 .sf/wiki/architecture.md create mode 100644 .sf/wiki/index.md create mode 100644 .sf/wiki/workflows.md diff --git a/.sf/wiki/architecture.md b/.sf/wiki/architecture.md new file mode 100644 index 000000000..320a3d69a --- /dev/null +++ b/.sf/wiki/architecture.md @@ -0,0 +1,67 @@ +# Architecture + +*Generated: 2026-05-11 | Source: ARCHITECTURE.md, docs/adr/* + +## Overview + +SF is a Purpose-to-Software Compiler (ADR-0000). It runs long-horizon coding work through the **Unified Operation Kernel (UOK)**: milestones → slices → tasks. A deterministic controller (not an LLM) reads canonical DB state and decides what to dispatch next. Each dispatch unit gets a fresh AI context. + +**Key invariant:** `sf.db` (SQLite via `node:sqlite`) is the single source of truth for all structured state. Markdown files are rendered projections or human-editable inputs, not authoritative sources. + +## Subsystems at a Glance + +| Subsystem | Path | Purpose | +|-----------|------|---------| +| CLI entry | `src/loader.ts` | Initializes resources, registers extension | +| Headless driver | `src/headless.ts` | Non-interactive mode, exit codes 0/1/10/11/12 | +| SF extension | `src/resources/extensions/sf/` | All SF autonomous flow logic | +| UOK orchestrator | `src/resources/extensions/sf/auto/` | Dispatch loop, lifecycle, planning | +| Context injection | `src/resources/extensions/sf/bootstrap/` | System prompt assembly | +| Prompt templates | `src/resources/extensions/sf/prompts/` | `.md` templates loaded at runtime | +| Agent core | `packages/coding-agent/` | Tools (bash, grep, read…), models, sessions, settings | +| LLM providers | `packages/ai/` | Anthropic, OpenAI, Google, Moonshot, etc. | +| Terminal UI | `packages/tui/` | Ink-based TUI | +| Daemon | `packages/daemon/` | Background process | +| RPC client | `packages/rpc-client/` | RPC adapter | +| Native addon | `packages/native/` | Rust N-API bindings (grep, glob, ps, highlight, ast, diff) | + +## Autonomous Dispatch Loop + +1. UOK reconciles DB-backed ledger + runtime diagnostics → typed state snapshot +2. Controller selects next unit (research → plan → implement → verify) from DB state +3. Fresh agent context started, task plan injected via `system-context.js` +4. Agent writes artifacts, commits, exits +5. UOK records completion/recovery, updates projections, repeats + +## System Context Assembly (per turn) + +`system-context.js` `buildBeforeAgentStartResult` assembles blocks in order: + +``` +escalation-policy → system.md → preferences → knowledge (DB) → ARCHITECTURE.md +→ tacit-knowledge → wiki (.sf/wiki/) → CODEBASE.md → code-intelligence +→ memories (DB) → new-skills → self-feedback → worktree/VCS → model identity +``` + +## State Layout + +**Tracked in git:** +- `.sf/milestones/` — plans, summaries (rendered projections) +- `.sf/PROJECT.md`, `.sf/REQUIREMENTS.md`, `.sf/DECISIONS.md`, etc. + +**Gitignored (runtime):** +- `.sf/sf.db*` — canonical SQLite state +- `.sf/wiki/` — generated + hand-curated wiki (this directory; **tracked in git**) +- `.sf/CODEBASE.md` — generated file map +- `.sf/runtime/`, `.sf/activity/`, `.sf/worktrees/`, etc. + +## Tools in bash tool + +Bash tool has a 120s default timeout (prevents autonomous mode hangs). Override via `settings.json → bash.defaultTimeoutSeconds`. Prefer dedicated tools: `grep`, `find`, `ls`, `read` over bash for file operations. + +## ADRs + +- `docs/adr/0000-purpose-to-software-compiler.md` — foundational product contract +- `docs/adr/0001-sf-state-directory.md` — `.sf/` state model +- `docs/adr/0004-capability-aware-routing.md` — model routing +- `docs/adr/0005-multi-model-provider-strategy.md` — provider strategy diff --git a/.sf/wiki/index.md b/.sf/wiki/index.md new file mode 100644 index 000000000..b112a30af --- /dev/null +++ b/.sf/wiki/index.md @@ -0,0 +1,53 @@ +# singularity-forge — Index + +*Generated: 2026-05-11 | Commit: see `git rev-parse HEAD` | Commands: see workflows.md* + +## What this repo is + +SF (Singularity Forge) is a standalone autonomous repo operator CLI built on the [Pi SDK](https://github.com/badlogic/pi-mono). It compiles purpose into software via a Purpose-to-Software Compiler model (ADR-0000). One command (`sf`) runs an LLM-powered agent that plans, implements, tests, commits, and advances through milestones autonomously. + +**Version:** 2.75.3 +**Package:** `singularity-forge` on npm +**Bin:** `sf` +**Node:** 26.1+ required (`.mise.toml`, `.nvmrc`, `.node-version` pin) + +## How to run it + +```bash +npm install -g singularity-forge@latest +sf # interactive TUI +sf headless ... # machine surface (JSON I/O) +``` + +Dev from source: +```bash +npm install +npm run build # full build (packages + web) +npm run build:core # packages + tsc + resources only +npm run dev # hot reload +npm run copy-resources # rebuild dist/resources/ (extension JS/prompts) +``` + +## Where to start + +| Goal | Start here | +|------|-----------| +| Understand the product | `README.md`, `VISION.md`, `ARCHITECTURE.md` | +| Understand SF's architecture decisions | `docs/adr/` | +| Understand spec-first TDD doctrine | `docs/SPEC_FIRST_TDD.md` | +| Understand the runtime state model | `.sf/PROJECT.md`, `AGENTS.md` | +| Find a subsystem | `subsystems.md` | +| Build/test commands | `workflows.md` | +| Understand SF autonomous mode | `src/resources/extensions/sf/` | +| Understand the Pi SDK packages | `packages/` | + +## Key files + +- `src/loader.ts` — CLI entry point (`dist/loader.js`, bin `sf`) +- `src/headless-context.ts` — headless context loading + auto-bootstrap +- `src/resources/extensions/sf/` — SF autonomous mode extension (compile via `copy-resources`) +- `packages/coding-agent/` — Pi agent core (tools, models, sessions, settings) +- `packages/ai/` — LLM provider adapters +- `packages/tui/` — Terminal UI +- `packages/daemon/` — Background daemon process +- `.sf/sf.db` — canonical structured state (SQLite, `node:sqlite`) diff --git a/.sf/wiki/workflows.md b/.sf/wiki/workflows.md new file mode 100644 index 000000000..a606d854e --- /dev/null +++ b/.sf/wiki/workflows.md @@ -0,0 +1,76 @@ +# Workflows + +*Generated: 2026-05-11 | Source: package.json, CONTRIBUTING.md, AGENTS.md* + +## Build + +```bash +npm run build # full (packages + tsc + resources + web) +npm run build:core # packages + tsc + resources only (no web) +npm run copy-resources # rebuild dist/resources/ ONLY (extension JS + prompts, ~60-90s) +``` + +**Critical:** editing `.ts` files in `src/resources/extensions/sf/` has no effect until `copy-resources` runs. The loader resolves from `dist/resources/extensions/sf/` (compiled JS). + +## Test + +```bash +npm test # unit + integration +npm run test:unit # vitest unit tests +npm run test:integration # integration tests +npm run test:coverage # coverage (thresholds: 40/40/20/20) +npm run test:smoke # smoke tests +npx vitest run --config vitest.config.ts # single file +npx vitest run --changed --config vitest.config.ts # affected only +``` + +Test files: `*.test.ts` and `*.test.mjs` under `packages/*/src/`, `src/tests/`, `src/resources/extensions/*/tests/`. + +## Lint + +```bash +npm run lint # Biome over src/ +npm run lint:fix # autofix +npm --prefix web run lint # ESLint over web/ +npx biome check --write # format specific files +npm run typecheck:extensions # tsc --noEmit for extensions +``` + +## Release + +```bash +npm run release:changelog # generate CHANGELOG entry +npm run release:bump # bump version +``` + +## SF Autonomous Flow + +``` +sf → TUI (interactive) +sf headless ... → machine surface (JSON I/O) +``` + +**Autonomous mode commands (headless):** +```bash +node dist/loader.js headless next # dispatch next ready unit +node dist/loader.js headless autonomous # run full autonomous loop +node dist/loader.js headless status # show current state +node dist/loader.js headless query # query DB state +``` + +**Note:** use `node dist/loader.js` not `npx sf` in this repo to avoid npm download hangs. + +## Dev workflow + +```bash +npm run dev # hot reload (tsc watch + copy-resources watch) +``` + +After editing prompts or extension JS, run `copy-resources` then test with headless. + +## Git conventions + +- Branch: `/` (feat/, fix/, chore/, refactor/, etc.) +- Commits: Conventional Commits format +- No merge commits — rebase onto main +- PRs must link an issue; one concern per PR diff --git a/src/headless-context.ts b/src/headless-context.ts index 6ab1f2be4..e38d4554e 100644 --- a/src/headless-context.ts +++ b/src/headless-context.ts @@ -271,6 +271,16 @@ function collectAutoBootstrapFiles(basePath: string): string[] { } } + // Include .sf/wiki/*.md pages — excluded from the general walk because + // .sf is in AUTO_BOOTSTRAP_EXCLUDED_DIRS, but wiki pages are high-value + // orientation context that should always be available to new agents. + for (const path of collectWikiFiles(basePath)) { + if (!seen.has(path)) { + seen.add(path); + files.push(path); + } + } + for (const path of walkMarkdownFiles(basePath)) { if (seen.has(path)) continue; seen.add(path); @@ -280,6 +290,20 @@ function collectAutoBootstrapFiles(basePath: string): string[] { return files; } +function collectWikiFiles(basePath: string): string[] { + const wikiDir = join(basePath, ".sf", "wiki"); + let entries: Dirent[]; + try { + entries = readdirSync(wikiDir, { withFileTypes: true }) as Dirent[]; + } catch { + return []; + } + return entries + .filter((e) => e.isFile() && e.name.toLowerCase().endsWith(".md")) + .sort((a, b) => a.name.localeCompare(b.name)) + .map((e) => join(wikiDir, e.name)); +} + function existsMarkdownFile(path: string): boolean { try { const stat = statSync(path); diff --git a/src/resources/extensions/sf/auto-bootstrap-context.js b/src/resources/extensions/sf/auto-bootstrap-context.js index 6ff087baf..9a1aa4305 100644 --- a/src/resources/extensions/sf/auto-bootstrap-context.js +++ b/src/resources/extensions/sf/auto-bootstrap-context.js @@ -184,6 +184,15 @@ function collectAutoBootstrapFiles(basePath) { files.push(path); } } + // Include .sf/wiki/*.md pages — excluded from the general walk because + // .sf is in AUTO_BOOTSTRAP_EXCLUDED_DIRS, but wiki pages are high-value + // orientation context that should always be available to new agents. + for (const path of collectWikiFiles(basePath)) { + if (!seen.has(path)) { + seen.add(path); + files.push(path); + } + } for (const path of walkMarkdownFiles(basePath)) { if (seen.has(path)) continue; seen.add(path); @@ -191,6 +200,19 @@ function collectAutoBootstrapFiles(basePath) { } return files; } +function collectWikiFiles(basePath) { + const wikiDir = join(basePath, ".sf", "wiki"); + let entries; + try { + entries = readdirSync(wikiDir, { withFileTypes: true }); + } catch { + return []; + } + return entries + .filter((e) => e.isFile() && e.name.toLowerCase().endsWith(".md")) + .sort((a, b) => a.name.localeCompare(b.name)) + .map((e) => join(wikiDir, e.name)); +} function existsMarkdownFile(path) { try { const stat = statSync(path); diff --git a/src/resources/extensions/sf/bootstrap/system-context.js b/src/resources/extensions/sf/bootstrap/system-context.js index 41e626de8..d7403b1da 100644 --- a/src/resources/extensions/sf/bootstrap/system-context.js +++ b/src/resources/extensions/sf/bootstrap/system-context.js @@ -1,4 +1,4 @@ -import { existsSync, readFileSync, statSync, unlinkSync } from "node:fs"; +import { existsSync, readdirSync, readFileSync, statSync, unlinkSync } from "node:fs"; import { sfHome } from '../sf-home.js'; import { join } from "node:path"; import { @@ -156,7 +156,7 @@ exhausted, not just because the next tier is faster. Tier 1 — Code lookup: - grep/find/ls for broad orientation; scoped sift / codebase_search for symbols, patterns, prior usages when Sift status is healthy for the repo - Read source files (Read tool, file paths from PLAN/CODEBASE) - - Inspect .sf/DECISIONS.md, .sf/KNOWLEDGE.md, docs/design-docs/, docs/records/ + - Inspect .sf/wiki/ (injected as WIKI block when present), .sf/DECISIONS.md, .sf/KNOWLEDGE.md, docs/design-docs/, docs/records/ - Check tests for documented behavior Tier 2 — External lookup (factual questions): @@ -214,6 +214,7 @@ export async function buildBeforeAgentStartResult(event, ctx) { ); const architectureBlock = loadArchitectureBlock(process.cwd()); const tacitKnowledgeBlock = loadTacitKnowledgeBlock(process.cwd()); + const wikiBlock = loadWikiBlock(process.cwd()); if (globalSizeKb > 4) { ctx.ui.notify( `SF: ~/.sf/agent/KNOWLEDGE.md is ${globalSizeKb.toFixed(1)}KB — consider trimming to keep system prompt lean.`, @@ -317,7 +318,7 @@ export async function buildBeforeAgentStartResult(event, ctx) { ? `\n\n[JUDGMENT LOG — autonomous mode]\nWhen you make a judgment call between alternatives at an ambiguous point, call log_decision with: decision, alternatives, reasoning, confidence. This lets the user review your reasoning at milestone close. It does NOT delay or block the work.` : ""; const selfFeedbackBlock = loadSelfFeedbackBlock(process.cwd()); - const fullSystem = `${event.systemPrompt}\n\n[SYSTEM CONTEXT — SF]\n\n${escalationPolicyBlock}${systemContent}${preferenceBlock}${knowledgeBlock}${architectureBlock}${tacitKnowledgeBlock}${codebaseBlock}${codeIntelligenceBlock}${memoryBlock}${newSkillsBlock}${selfFeedbackBlock}${worktreeBlock}${repositoryVcsBlock}${modelIdentityBlock}${subagentModelBlock}${judgmentLogBlock}`; + const fullSystem = `${event.systemPrompt}\n\n[SYSTEM CONTEXT — SF]\n\n${escalationPolicyBlock}${systemContent}${preferenceBlock}${knowledgeBlock}${architectureBlock}${tacitKnowledgeBlock}${wikiBlock}${codebaseBlock}${codeIntelligenceBlock}${memoryBlock}${newSkillsBlock}${selfFeedbackBlock}${worktreeBlock}${repositoryVcsBlock}${modelIdentityBlock}${subagentModelBlock}${judgmentLogBlock}`; stopContextTimer({ systemPromptSize: fullSystem.length, injectionSize: injection?.length ?? forensicsInjection?.length ?? 0, @@ -484,6 +485,52 @@ export function loadTacitKnowledgeBlock(cwd) { if (nonGoals) parts.push(`\n## Non-goals\n\n${nonGoals}`); return `\n\n${parts.join("\n")}`; } +const WIKI_FILE_MAX_CHARS = 4_000; +const WIKI_TOTAL_MAX_CHARS = 16_000; +/** + * Load .sf/wiki/*.md files into a single context block. Each file is capped at + * 4 000 chars; total block is capped at 16 000 chars. Files are sorted + * alphabetically. Subdirectories are skipped. + * + * The wiki is the primary place for project-specific reference docs, ADRs, + * backlog summaries, and architecture notes that should persist across sessions. + */ +export function loadWikiBlock(cwd) { + const wikiDir = join(cwd, ".sf", "wiki"); + if (!existsSync(wikiDir)) return ""; + let entries; + try { + entries = readdirSync(wikiDir, { withFileTypes: true }); + } catch { + return ""; + } + const mdFiles = entries + .filter((e) => e.isFile() && e.name.endsWith(".md")) + .map((e) => e.name) + .sort(); + if (mdFiles.length === 0) return ""; + const sections = []; + let totalChars = 0; + for (const filename of mdFiles) { + if (totalChars >= WIKI_TOTAL_MAX_CHARS) { + sections.push( + `*(additional wiki files omitted — see .sf/wiki/ for full content)*`, + ); + break; + } + const raw = cachedReadFile(join(wikiDir, filename))?.trim() ?? ""; + if (!raw) continue; + const content = + raw.length > WIKI_FILE_MAX_CHARS + ? raw.slice(0, WIKI_FILE_MAX_CHARS) + + `\n\n*(truncated — see .sf/wiki/${filename} for full content)*` + : raw; + sections.push(`### ${filename}\n\n${content}`); + totalChars += content.length; + } + if (sections.length === 0) return ""; + return `\n\n[WIKI — Project reference docs (.sf/wiki/)]\n\n${sections.join("\n\n---\n\n")}`; +} /** * Load ARCHITECTURE.md from the project root into context. Capped at 8 000 chars * to avoid bloating every request — full file is always readable on disk. diff --git a/src/resources/extensions/sf/prompts/system.md b/src/resources/extensions/sf/prompts/system.md index 6c9de441f..bc083dbe3 100644 --- a/src/resources/extensions/sf/prompts/system.md +++ b/src/resources/extensions/sf/prompts/system.md @@ -77,6 +77,12 @@ Titles live inside file content (headings, frontmatter), not in file or director DECISIONS.md (append-only register of architectural and pattern decisions) KNOWLEDGE.md (append-only register of project-specific rules, patterns, and lessons learned) CODEBASE.md (generated fallback codebase map cache — auto-refreshed when tracked files change) + wiki/ (generated + hand-curated reference wiki — tracked in git; use sf-wiki skill to generate) + index.md (what this repo is, how to run it, where to start) + architecture.md (major subsystems and data/control flow) + workflows.md (build, test, release, autonomous/SF flows) + subsystems.md (table of subsystem, path, purpose, owner, tests) + glossary.md (project terms only) OVERRIDES.md (user-issued overrides that supersede plan content via /steer) QUEUE.md (append-only log of queued milestones via /queue) STATE.md @@ -120,6 +126,7 @@ In all modes, slices commit sequentially on the active branch; there are no per- - **DECISIONS.md** is an append-only register of architectural and pattern decisions - read it during planning/research, append to it during execution when a meaningful decision is made - **KNOWLEDGE.md** is an append-only register of project-specific rules, patterns, and lessons learned. Read it at the start of every unit. Append to it when you discover a recurring issue, a non-obvious pattern, or a rule that future agents should follow. - **CODEBASE.md** is a generated fallback snapshot of the tracked repository. SF may inject it when available, but healthy Sift is the preferred live code index. Use CODEBASE only when Sift is unavailable, cold, degraded, or when you need a durable overview. Use `/codebase update` only when you need to force an immediate refresh. +- **wiki/** (`.sf/wiki/`) contains generated and hand-curated codebase reference pages — injected automatically when present. Contains `index.md`, `architecture.md`, `workflows.md`, `subsystems.md`, and `glossary.md`. Read it at the start of any planning or research unit for fast repo orientation. Generate or refresh with the `sf-wiki` skill; pages are tracked in git so hand edits persist. - **CONTEXT.md** files (milestone or slice level) capture the brief — scope, goals, constraints, and key decisions from discussion. When present, they are the authoritative source for what a milestone or slice is trying to achieve. Read them before planning or executing. - **Milestones** are major project phases (M001, M002, ...) - **Slices** are demoable vertical increments (S01, S02, ...) ordered by risk. After each slice completes, the roadmap is reassessed before the next slice begins.