singularity-forge/docs/dev/architecture.md

# Architecture Overview

SF is a purpose-to-software compiler built as a TypeScript application on the [Pi SDK](https://github.com/badlogic/pi-mono). It embeds the Pi coding agent as an execution surface, but SF's core contract is stronger: convert bounded intent into PDD fields, research assumptions, generate tests or executable evidence first, implement the smallest satisfying change, verify, and retain useful memory.

The foundational decision is [ADR-0000: SF Is a Purpose-to-Software Compiler](../adr/0000-purpose-to-software-compiler.md).

## System Structure

```
sf (CLI binary)
  └─ loader.ts          Sets PI_PACKAGE_DIR, SF env vars, dynamic-imports cli.ts
      └─ cli.ts         Wires SDK managers, loads extensions, starts InteractiveMode
          ├─ onboarding.ts   First-run setup wizard (LLM provider + tool keys)
          ├─ wizard.ts       Env hydration from stored auth.json credentials
          ├─ app-paths.ts    ~/.sf/agent/, ~/.sf/sessions/, auth.json
          ├─ resource-loader.ts  Syncs bundled extensions + agents to ~/.sf/agent/
          └─ src/resources/
              ├─ extensions/sf/    Core SF extension
              ├─ extensions/...     23 supporting extensions
              ├─ agents/            scout, researcher, worker
              ├─ AGENTS.md          Agent routing instructions
              └─ SF-WORKFLOW.md    Manual bootstrap protocol

sf headless              Headless mode — CI/cron orchestration via RPC child process

vscode-extension/         VS Code extension — chat participant (@sf), sidebar dashboard, RPC integration
```

## Key Design Decisions

### State Lives on Disk

Structured `.sf` state is the runtime source of truth. For migrated milestones, `.sf/sf.db` is authoritative for hierarchy, `sequence` priority/order, validation assessments, gates, UOK lifecycle, and outcome ledgers. Markdown and JSON planning files are generated views, exports, or explicit recovery/import inputs; normal auto mode does not fall back to them when the DB exists and opens. No in-memory state survives across sessions. This enables crash recovery, multi-terminal steering, and session resumption.

### Two-File Loader Pattern

`loader.ts` sets all environment variables with zero SDK imports, then dynamically imports `cli.ts` which does static SDK imports. This ensures `PI_PACKAGE_DIR` is set before any SDK code evaluates.

### `pkg/` Shim Directory

`PI_PACKAGE_DIR` points to `pkg/` (not project root) to avoid Pi's theme resolution colliding with SF's `src/` directory. Contains only `piConfig` and theme assets.

### Always-Overwrite Sync

Bundled extensions and agents are synced to `~/.sf/agent/` on every launch, not just first run. This means `npm update -g` takes effect immediately.

### Lazy Provider Loading

LLM provider SDKs (Anthropic, OpenAI, Google, etc.) are lazy-loaded on first use rather than imported at startup. This significantly reduces cold-start time — only the provider you actually connect to gets loaded.

### Fresh Session Per Unit

Every dispatch creates a new agent session. The LLM starts with a clean context window containing only the pre-inlined artifacts it needs. This prevents quality degradation from context accumulation.

## Bundled Extensions

| Extension | What It Provides |
|-----------|-----------------|
| **SF** | Core workflow engine — UOK-governed auto mode, commands, dashboard |
| **Browser Tools** | Playwright-based browser automation — navigation, forms, screenshots, PDF export, device emulation, visual regression, structured data extraction, route mocking, accessibility tree inspection, and semantic actions |
| **Search the Web** | Brave Search, Tavily, or Jina page extraction |
| **Google Search** | Gemini-powered web search with AI-synthesized answers |
| **Context7** | Up-to-date library/framework documentation |
| **Background Shell** | Long-running process management with readiness detection |
| **Subagent** | Delegated tasks with isolated context windows |
| **MCP Client** | Native MCP server integration via @modelcontextprotocol/sdk |
| **Voice** | Real-time speech-to-text on Linux |
| **Slash Commands** | Custom command creation |
| **LSP** | Language Server Protocol — diagnostics, definitions, references, hover, rename |
| **Ask User Questions** | Structured user input with single/multi-select |
| **Secure Env Collect** | Masked secret collection |
| **Async Jobs** | Background command execution with `async_bash`, `await_job`, `cancel_job` |
| **Remote Questions** | Discord, Slack, and Telegram integration for headless question routing |
| **TTSR** | Tool-triggered system rules — conditional context injection based on tool usage |
| **Universal Config** | Discovery of external tool configurations for import or migration |
| **AWS Auth** | AWS credential management and authentication |
| **cmux** | Context multiplexing for multi-session coordination |
| **GitHub Sync** | GitHub issue and PR synchronization |
| **Ollama** | Local Ollama model integration |
| **Shared** | Shared utilities across extensions |

## Bundled Agents

| Agent | Role |
|-------|------|
| **Scout** | Fast codebase recon — compressed context for handoff |
| **Researcher** | Web research — finds and synthesizes current information |
| **Worker** | General-purpose execution in an isolated context window |

## Native Engine

Performance-critical operations use a Rust N-API engine:

- **grep** — ripgrep-backed content search
- **glob** — gitignore-aware file discovery
- **ps** — cross-platform process tree management
- **highlight** — syntect-based syntax highlighting
- **ast** — structural code search via ast-grep
- **diff** — fuzzy text matching and unified diff generation
- **text** — ANSI-aware text measurement and wrapping
- **html** — HTML-to-Markdown conversion
- **image** — decode, encode, resize images
- **fd** — fuzzy file path discovery
- **clipboard** — native clipboard access
- **git** — libgit2-backed git read operations (v2.16+)
- **parser** — SF file parsing and frontmatter extraction

## Dispatch Pipeline

The auto mode dispatch pipeline:

```
1.  Reconcile canonical project state, the UOK run ledger, and runtime projections
2.  Determine next unit type and ID
3.  Classify complexity → select model tier
4.  Apply budget pressure adjustments
5.  Check routing history for adaptive adjustments
6.  Dynamic model routing (if enabled) → select cheapest model for tier
7.  Resolve effective model (with fallbacks)
8.  Check pending captures → triage if needed
9.  Build dispatch prompt (applying inline level compression)
10. Create fresh agent session
11. Inject prompt and let LLM execute
12. On completion: snapshot metrics, verify artifacts, persist UOK ledger state, update projections
13. Loop to step 1
```

Phase skipping (from token profile) gates steps 2-3: if a phase is skipped, the corresponding unit type is never dispatched.

## Key Modules (v2.67)

| Module | Purpose |
|--------|---------|
| `auto.ts` | Auto-mode orchestration over the UOK lifecycle and dispatch loop |
| `auto/session.ts` | `AutoSession` class — all mutable auto-mode state in one encapsulated instance |
| `auto-dispatch.ts` | Declarative dispatch table (phase → unit mapping) |
| `auto-idempotency.ts` | Completed-key checks, skip loop detection, key eviction |
| `auto-stuck-detection.ts` | Stuck loop recovery and unit retry escalation |
| `auto-start.ts` | Fresh-start bootstrap — git/state init, crash lock detection, worktree setup |
| `auto-post-unit.ts` | Post-unit processing — commit, doctor, state rebuild, hooks |
| `auto-verification.ts` | Post-unit verification gate (lint/test/typecheck with auto-fix retries) |
| `auto-prompts.ts` | Prompt builders with inline level compression |
| `auto-worktree.ts` | Worktree lifecycle (create, enter, merge, teardown) |
| `auto-recovery.ts` | Expected artifact resolution, completed-key persistence, self-healing |
| `auto-timeout-recovery.ts` | Timed-out unit recovery and continuation |
| `auto-timers.ts` | Unit supervision — soft/idle/hard timeouts, continue-here monitor |
| `complexity-classifier.ts` | Unit complexity classification (light/standard/heavy) |
| `model-router.ts` | Dynamic model routing with cost-aware selection |
| `model-cost-table.ts` | Built-in per-model cost data for cross-provider comparison |
| `routing-history.ts` | Adaptive learning from routing outcomes |
| `captures.ts` | Fire-and-forget thought capture and triage classification |
| `triage-resolution.ts` | Capture resolution (inject, defer, replan, quick-task) |
| `visualizer-overlay.ts` | Workflow visualizer TUI overlay |
| `visualizer-data.ts` | Data loading for visualizer tabs |
| `visualizer-views.ts` | Tab renderers (progress, deps, metrics, timeline, discussion status) |
| `metrics.ts` | Token and cost tracking ledger |
| `state.ts` | DB-authoritative state derivation with filesystem fallback only for unmigrated/recovery planning artifacts |
| `session-lock.ts` | OS-level exclusive session locking (proper-lockfile) |
| `crash-recovery.ts` | Lock file management for crash detection and recovery |
| `preferences.ts` | Preference loading, merging, validation |
| `git-service.ts` | Git operations — commit, merge, worktree sync, completed-units cross-boundary sync |
| `unit-id.ts` | Centralized `parseUnitId()` — milestone/slice/task extraction from unit IDs |
| `error-utils.ts` | `getErrorMessage()` — unified error-to-string conversion |
| `roadmap-slices.ts` | Roadmap parser with prose fallback for LLM-generated variants |
| `memory-extractor.ts` | Extract reusable knowledge from session transcripts |
| `memory-store.ts` | Persistent memory store for cross-session knowledge; query-aware ranking |
| `memory-embeddings.ts` | Vector storage + cosine ranking + agent_end backfill driver |
| `memory-embeddings-llm-gateway.ts` | OpenAI-shaped /v1/embeddings + /v1/rerank adapter for the inference-fabric llm-gateway (env-driven; opt-in via `SF_LLM_GATEWAY_KEY`) |
| `memory-relations.ts` | Knowledge-graph edges between memories (typed relations: `related_to`, `depends_on`, `contradicts`, `elaborates`, `supersedes`). Memory-extraction batches auto-link co-extracted memories with `related_to` (confidence 0.5). `getRelevantMemoriesRanked` walks the cosine top-N's edges and applies a one-pass score boost (intra-pool, damping=0.4) so cohort memories surface together. |
| `memory-source-store.ts` | Raw source rows preserved separately from extracted memories (idempotent re-extraction) |
| `queue-order.ts` | Milestone queue ordering |
| `context-masker.ts` | Context masking for model routing optimization |
| `phase-anchor.ts` | Phase anchoring for dispatch pipeline |
| `slice-parallel-orchestrator.ts` | Slice-level parallelism with dependency-aware dispatch |
| `slice-parallel-eligibility.ts` | Slice parallel eligibility checks |
| `slice-parallel-conflict.ts` | Slice parallel conflict detection |
| `preferences-models.ts` | Model preferences configuration |
| `preferences-validation.ts` | Preferences validation |
| `preferences-types.ts` | Preferences type definitions |