singularity/singularity-forge
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
singularity-forge/docs/dev/ADR-009-IMPLEMENTATION-PLAN.md
ace-pm b29c12d5e5 refactor(native): rename gsd_parser.rs to forge_parser.rs 
Final rebrand: rename remaining Rust source file to complete the gsd → forge
transition. All parser references already use forge_parser after earlier commits.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 14:58:21 +02:00

14 KiB
Raw Blame History

ADR-009 Implementation Plan

Related ADR: ADR-009-orchestration-kernel-refactor.md
Status: Draft
Date: 2026-04-14
Target Window: 8-10 waves (incremental, no big-bang rewrite)

Objective

Implement ADR-009 by migrating SF orchestration internals to a Unified Orchestration Kernel (UOK) with six control planes:

  1. Plan
  2. Execution
  3. Model
  4. Gate
  5. GitOps
  6. Audit

without breaking existing CLI/web/MCP workflows.

The first production-safe outcome is:

  • existing auto-mode behavior remains stable
  • new kernel contracts exist behind feature flags
  • every turn is traceable with deterministic gate outcomes

Non-Goals

  • Rewriting user-facing command surfaces
  • Replacing all legacy modules in a single PR
  • Introducing new provider auth flows that bypass existing compliance boundaries
  • Forcing burn-max behavior as default

Constraints

  • Maintain current runtime compatibility and defaults
  • Preserve existing state-on-disk and DB-backed transition model
  • Keep provider-agnostic behavior while enforcing provider-specific policy constraints
  • All migration steps must be reversible behind flags
  • High-risk changes require parity tests against existing behavior

Program Structure

Implementation is organized into parallel workstreams and executed in waves.

Workstream A: Kernel Contracts and Orchestrator Spine

Goal: define typed contracts and a new orchestration spine without changing behavior.

Primary targets:

  • src/resources/extensions/sf/auto.ts
  • src/resources/extensions/sf/auto/loop.ts
  • src/resources/extensions/sf/auto/types.ts
  • src/resources/extensions/sf/auto/session.ts

Deliverables:

  • TurnContract and TurnResult types
  • GateResult envelope
  • kernel entrypoint that wraps current dispatch loop via adapter

Workstream B: Gate Plane

Goal: normalize all checks into a unified gate runner.

Primary targets:

  • src/resources/extensions/sf/verification-gate.ts
  • src/resources/extensions/sf/auto-verification.ts
  • src/resources/extensions/sf/pre-execution-checks.ts
  • src/resources/extensions/sf/post-execution-checks.ts
  • src/resources/extensions/sf/milestone-validation-gates.ts

Deliverables:

  • unified gate registry and execution API
  • deterministic failure classes and retry policies
  • explicit terminal status persistence

Workstream C: Model Plane + Policy Engine

Goal: enable any-model-any-phase through requirement-based selection plus policy filtering.

Primary targets:

  • src/resources/extensions/sf/model-router.ts
  • src/resources/extensions/sf/auto-model-selection.ts
  • src/resources/extensions/sf/preferences-models.ts
  • src/resources/extensions/sf/model-cost-table.ts
  • src/resources/extensions/sf/custom-execution-policy.ts

Deliverables:

  • requirement vector builder for units
  • policy filter before capability scoring
  • new burn-max profile
  • policy decision audit events

Workstream D: Execution Graph (Agents/Subagents/Parallel/Teams)

Goal: move to one DAG scheduler contract.

Primary targets:

  • src/resources/extensions/sf/reactive-graph.ts
  • src/resources/extensions/sf/slice-parallel-orchestrator.ts
  • src/resources/extensions/sf/parallel-orchestrator.ts
  • src/resources/extensions/sf/graph.ts
  • src/resources/extensions/sf/unit-runtime.ts

Deliverables:

  • typed node kinds (unit, hook, subagent, team-worker, verification, reprocess)
  • shared dependency/conflict resolver
  • scheduler adapter for current parallel and reactive paths

Workstream E: GitOps Transaction Layer

Goal: guarantee git action and metadata record per turn.

Primary targets:

  • src/resources/extensions/sf/git-service.ts
  • src/resources/extensions/sf/auto-post-unit.ts
  • src/resources/extensions/sf/auto-unit-closeout.ts
  • src/resources/extensions/sf/auto-worktree.ts

Deliverables:

  • turn-start -> stage -> checkpoint -> publish -> record transaction API
  • configurable turn action mode (commit|snapshot|status-only)
  • closeout gate integration for git failures

Workstream F: Unified Audit Plane

Goal: unify journal/activity/metrics into a causal event model.

Primary targets:

  • src/resources/extensions/sf/journal.ts
  • src/resources/extensions/sf/activity-log.ts
  • src/resources/extensions/sf/metrics.ts
  • src/resources/extensions/sf/workflow-logger.ts
  • src/resources/extensions/sf/sf-db.ts

Deliverables:

  • common AuditEventEnvelope
  • trace/turn IDs on all events
  • append-only JSONL raw log + DB projection index

Workstream G: Plan Plane v2

Goal: formal multi-round clarify/research/draft/compile flow.

Primary targets:

  • src/resources/extensions/sf/guided-flow.ts
  • src/resources/extensions/sf/preparation.ts
  • src/resources/extensions/sf/auto/phases.ts
  • src/resources/extensions/sf/auto-prompts.ts
  • prompt templates under src/resources/extensions/sf/prompts/

Deliverables:

  • bounded multi-round question loop
  • plan compile step producing executable unit graph
  • plan gate fail-closed behavior

Wave Plan (Execution Order)

Wave 0: Baseline and Flag Scaffolding

Purpose: establish safe rollout controls and baseline telemetry.

Tasks:

  • Add feature flags:
    • uok.enabled
    • uok.gates.enabled
    • uok.model_policy.enabled
    • uok.execution_graph.enabled
    • uok.gitops.enabled
    • uok.audit_unified.enabled
    • uok.plan_v2.enabled
  • Add no-op kernel wrapper around current auto loop
  • Add baseline metrics for parity comparison

Exit criteria:

  • zero behavior change with all flags off
  • parity telemetry collected for existing loop

Verification:

  • npm run typecheck:extensions
  • npm run test:unit

Wave 1: Contract Extraction

Purpose: create stable internal API boundaries.

Tasks:

  • Introduce:
    • TurnContract
    • UnitExecutionContext
    • GateResult
    • FailureClass
    • TurnCloseoutRecord
  • Adapter layer from legacy auto loop into contracts
  • Add contract fixtures and serialization tests

Exit criteria:

  • current auto dispatch runs through adapter path without behavior change
  • all turn outcomes represented in structured result type

Verification:

  • targeted tests in src/resources/extensions/sf/tests/*auto*
  • npm run test:unit

Wave 2: Gate Plane Unification

Purpose: centralize pre/in/post checks and retries.

Tasks:

  • Build gate-runner and gate registry
  • Port existing checks into registered gates:
    • policy/input/execution/artifact/verification/closeout
  • Implement deterministic retry matrix by failure class

Exit criteria:

  • every unit passes through gate runner
  • explicit gate result persisted for pass/fail/retry/manual-attention

Verification:

  • extend verification-gate.test.ts
  • extend validation-gate-patterns.test.ts
  • add integration tests for retry escalation

Wave 3: Model Plane + Policy Filter

Purpose: enable requirement-based selection constrained by policy.

Tasks:

  • Add requirement extraction from unit metadata
  • Insert policy filter before model scoring
  • Add burn-max token profile wiring
  • Emit model policy allow/deny events

Exit criteria:

  • units can select any eligible model across phases
  • policy-denied routes fail before dispatch
  • fallback chains remain deterministic

Verification:

  • extend model-cost-table.test.ts
  • extend model routing tests (interactive-routing-bypass, tool-compatibility, related router suites)
  • add policy denial regression tests

Wave 4: Execution Graph Scheduler

Purpose: unify hooks/subagents/parallel/team work under one scheduler contract.

Tasks:

  • Introduce graph scheduler facade
  • Map reactive execution nodes to shared node model
  • Map slice/milestone parallel orchestrators onto scheduler
  • Add file IO conflict lock integration

Exit criteria:

  • same task set can execute in deterministic single-worker or parallel graph mode
  • no deadlock under known reactive/parallel fixtures

Verification:

  • slice-parallel-orchestrator.test.ts
  • slice-parallel-conflict.test.ts
  • sidecar-queue.test.ts
  • integration: src/resources/extensions/sf/tests/integration/*.test.ts

Wave 5: GitOps Transactions Per Turn

Purpose: enforce turn-level git actions and closeout discipline.

Tasks:

  • Implement turn transaction API
  • Wire turn transactions into auto closeout path
  • Add configurable turn_action and turn_push semantics
  • Persist git transaction metadata into audit stream

Exit criteria:

  • each turn has a git transaction record
  • blocked git states surface as closeout gate failures

Verification:

  • git-service integration tests
  • worktree-related integration suites
  • closeout and merge regression suites

Wave 6: Unified Audit Plane

Purpose: converge logging/metrics/journal into one causal model.

Tasks:

  • Define AuditEventEnvelope schema
  • Add traceId, turnId, causedBy to event emitters
  • Write projection pipeline into DB index tables
  • Maintain append-only raw JSONL logs

Exit criteria:

  • action-level traceability across model/tool/git/gate/test events
  • legacy readers remain functional through compatibility projection

Verification:

  • workflow-logger*.test.ts
  • workflow-events.test.ts
  • journal and metrics regression tests

Wave 7: Plan Plane v2

Purpose: deliver full multi-round planning and compile-to-unit graph.

Tasks:

  • Implement bounded clarify rounds
  • Add explicit research synthesis stage
  • Add plan compile stage with dependency graph output
  • Add plan gate with fail-closed checks

Exit criteria:

  • full roadmap and unit graph produced before execution begins (when enabled)
  • invalid plans cannot proceed to execution

Verification:

  • prompt and plan parsing tests
  • planning tool tests (plan-milestone, plan-slice, plan-task)
  • discuss/guided flow regression tests

Wave 8: Legacy Branch Retirement + Default Flip

Purpose: reduce maintenance burden and enable UOK as default.

Tasks:

  • remove superseded code paths in auto.ts, auto-phases, and legacy closeout paths
  • keep legacy fallback behind emergency flag for one release window
  • update docs and preferences reference

Exit criteria:

  • UOK default in stable channel
  • no critical parity regressions in one full release cycle

Verification:

  • full npm test
  • smoke + integration suites
  • targeted manual UAT for CLI/web/headless

Testing and Validation Matrix

1. Unit

  • contract serialization
  • gate runner behavior by failure class
  • model policy filter decisions
  • git transaction state machine
  • event envelope schema validation

2. Integration

  • auto dispatch across plan/execute/complete/reassess/uat
  • worktree/branch/none isolation behaviors
  • parallel and reactive execution parity
  • policy-denied dispatch fast-fail

3. End-to-End

  • greenfield milestone from discuss -> plan -> execute -> complete -> merge
  • failure reprocessing (test failure, tool failure, model failure)
  • full audit trace reconstruction by traceId
  • provider compliance scenarios (allowed vs denied paths)

4. Parity Harness

  • replay selected historical workflows against legacy and UOK paths
  • compare:
    • state transitions
    • produced artifacts
    • gate decisions
    • commit outcomes

Rollout Strategy

Stages

  1. Internal dogfood with flags on
  2. Beta cohort opt-in via project preference
  3. General availability with flags default-on
  4. Legacy fallback removed after stability window

Safety Controls

  • runtime kill-switch for each plane
  • release-note explicit migration warnings
  • auto-rollback trigger on critical regressions (gates, git integrity, state corruption)

Data and Schema Changes

Expected schema additions:

  • audit projection tables in sf.db
  • gate result persistence tables
  • turn transaction metadata

Rules:

  • additive migrations only until Wave 8
  • keep backwards-compatible readers during migration window

Dependencies

  1. Stable contract definitions before gate/model/scheduler rewires
  2. Gate plane before gitops hard enforcement
  3. Model policy engine before enabling any-model-any-phase by default
  4. Audit envelope before legacy logger removal
  5. Plan v2 before enforcing front-loaded planning defaults

Risk Register

Risk 1: Hidden Coupling in Auto Loop

Impact: migration bugs due to implicit side effects.
Mitigation: adapter-first extraction and parity harness before path switch.

Risk 2: Parallel Deadlocks

Impact: blocked runs or inconsistent state.
Mitigation: graph-level deadlock checks, IO lock tests, staged rollout behind flags.

Risk 3: Git Noise / Team Workflow Friction

Impact: commit churn and review overhead.
Mitigation: milestone squash defaults and configurable turn transaction modes.

Risk 4: Policy Drift Across Providers

Impact: compliance regressions.
Mitigation: provider policy registry tests and release checklist gates.

Risk 5: Telemetry Volume Growth

Impact: storage/perf pressure in long-running projects.
Mitigation: append-only raw + indexed projection + retention policies.

Definition of Done (ADR-009)

ADR-009 is complete when all are true:

  1. UOK path is default and stable.
  2. All units execute through unified gate runner.
  3. Model selection supports any eligible model in any phase with policy enforcement.
  4. Hooks/agents/subagents/parallel/team execution runs through one scheduler contract.
  5. Turn-level git transaction record exists for every executed turn.
  6. Unified audit events provide causal traceability across orchestration, model, tool, git, and test actions.
  7. Plan v2 can produce a complete unit graph with fail-closed plan gate.
  8. burn-max profile is available and policy-safe.
  9. Legacy orchestration branches are retired or behind emergency-only fallback.
  10. CLI/web/headless behavior remains user-compatible.

Recommended Immediate Next Tasks (Week 1)

  1. Add Wave 0 feature flags and default-off wiring.
  2. Introduce contract types and adapter shell (Wave 1 scaffolding).
  3. Add parity telemetry capture for legacy loop baseline.
  4. Land initial tests for contract serialization and turn result envelopes.