singularity-forge/docs/dev/PRD-branchless-worktree-architecture.md

# PRD: Branchless Worktree Architecture

**Author:** Lex Christopherson
**Date:** 2026-03-15
**ADR:** [ADR-001-branchless-worktree-architecture.md](./ADR-001-branchless-worktree-architecture.md)
**Priority:** Critical — blocks reliable auto-mode operation

---

## Problem Statement

SF's auto-mode is unreliable. Users experience:

1. **Infinite loop detection failures** — the agent writes planning artifacts on slice branches that become invisible after branch switching, causing `verifyExpectedArtifact()` to fail repeatedly. Auto-mode burns budget retrying the same unit 3-6 times before hard-stopping. This is the #1 user complaint.

2. **State corruption across branches** — `.sf/` planning artifacts (roadmaps, plans, decisions) are gitignored but branch-specific. Multiple branches sharing a single `.sf/` directory clobber each other's state. Users see wrong milestones marked complete, wrong roadmaps loaded, and auto-mode starting from the wrong phase.

3. **Excessive complexity** — 770+ lines of merge, conflict resolution, branch switching, and self-healing code exist solely to manage slice branches inside worktrees. This code has required 15+ bug fixes across versions and remains the primary source of auto-mode failures.

These problems are architectural. They cannot be fixed by patching individual symptoms.

## Vision

Auto-mode uses git worktrees for isolation and sequential commits for history. No branch switching. No merge conflicts within a worktree. Planning artifacts are tracked in git and travel with the branch. The git layer is so simple it can't break.

## Success Criteria

| Criterion | Measurement |
|-----------|-------------|
| Zero loop detection failures from branch visibility | No `verifyExpectedArtifact()` failures caused by branch mismatch in 50 consecutive auto-mode runs |
| Zero `.sf/` state corruption | Manual worktrees created via `git worktree add` have correct `.sf/` state without any SF-specific initialization |
| Code deletion | Net removal of ≥500 lines of merge/conflict/branch-switching code |
| Test simplification | Removal or simplification of ≥6 merge-specific test files |
| Backwards compatibility | Existing projects with `sf/M001/S01` slice branches continue to work (read-only; new work uses new model) |
| No new git primitives | The implementation uses only: worktrees, commits, squash-merge. No new branch types, merge strategies, or conflict resolution. |

## Non-Goals

- Parallel slice execution within a single worktree (if needed later, use separate worktrees)
- Changing how milestones relate to `main` (squash-merge stays)
- Modifying the dispatch unit types or state machine (except removing `fix-merge`)
- Changing the worktree-manager.ts manual worktree API (`/worktree` command)

## Current Architecture

### Branch Model (M003, v2.13.0)

```
main
  └─ milestone/M001 (worktree at .sf/worktrees/M001/)
       ├─ sf/M001/S01 (slice branch — code + .sf/ artifacts)
       │   └── merge --no-ff → milestone/M001
       ├─ sf/M001/S02
       │   └── merge --no-ff → milestone/M001
       └── squash merge → main
```

### Data Flow

```
Agent writes file → on slice branch → handleAgentEnd → auto-commit on slice branch
→ switch to milestone branch → verifyExpectedArtifact → FILE NOT FOUND (it's on slice branch)
→ loop counter++ → retry → same result → HARD STOP
```

### Code Involved

| File | Lines | Purpose |
|------|-------|---------|
| `auto-worktree.ts` | 512 | Worktree lifecycle + slice→milestone merge |
| `git-service.ts` | 915 | Branch creation, switching, merge with conflict resolution |
| `git-self-heal.ts` | 198 | Merge failure recovery |
| `auto.ts` | ~150 lines | Merge dispatch guards, fix-merge routing, branch-mode vs worktree-mode branching |
| `worktree.ts` | ~40 lines | Slice branch delegates |
| 11 test files | ~2000 lines | Merge/branch/worktree test coverage |

### `.sf/` Tracking (Current — Contradictory)

- `.gitignore` line 52: `.sf/` — ignores everything
- `smartStage()` lines 338-349: force-adds `SF_DURABLE_PATHS` — tracks milestones/, DECISIONS.md, PROJECT.md, REQUIREMENTS.md, QUEUE.md
- Result: `.sf/milestones/` is partially tracked on some branches, fully ignored on others. The code fights the config.

## Proposed Architecture

### Branch Model

```
main
  └─ milestone/M001 (worktree at .sf/worktrees/M001/)
       │
       commit: feat(M001): context + roadmap
       commit: feat(M001/S01): research
       commit: feat(M001/S01): plan
       commit: feat(M001/S01/T01): implement auth service
       commit: feat(M001/S01/T02): implement auth tests
       commit: feat(M001/S01): summary + UAT
       commit: docs(M001): reassess roadmap after S01
       commit: feat(M001/S02): research
       commit: feat(M001/S02): plan
       commit: ...
       commit: feat(M001): milestone complete
       │
       └── squash merge → main
```

One branch. Sequential commits. No merges within the worktree.

### Data Flow

```
Agent writes file → on milestone branch → handleAgentEnd → auto-commit on milestone branch
→ verifyExpectedArtifact → FILE FOUND (same branch) → persist completion → next dispatch
```

### `.sf/` Tracking (Proposed — Coherent)

**Tracked (travels with branch):**
```
.sf/milestones/**/*.md    (except CONTINUE markers)
.sf/milestones/**/*.json  (META.json integration records)
.sf/PROJECT.md
.sf/DECISIONS.md
.sf/REQUIREMENTS.md
.sf/QUEUE.md
```

**Gitignored (ephemeral):**
```
.sf/auto.lock
.sf/completed-units.json
.sf/STATE.md
.sf/metrics.json
.sf/sf.db
.sf/activity/
.sf/runtime/
.sf/worktrees/
.sf/DISCUSSION-MANIFEST.json
.sf/milestones/**/*-CONTINUE.md
.sf/milestones/**/continue.md
```

### Why This Works

| Problem | How It's Solved |
|---------|----------------|
| Artifact invisibility after branch switch | No branch switching. Artifacts commit on the one branch. |
| `.sf/` state clobbering | Artifacts tracked in git. Each branch carries its own `.sf/`. `git worktree add` and `git checkout` give correct state. |
| Merge conflict complexity | No merges within a worktree. Only merge is milestone→main (squash). |
| Manual worktree initialization | Tracked artifacts are checked out with the branch. No SF-specific bootstrap needed. |
| Dual isolation mode maintenance | Single mode: worktree. Branch-mode (`git.isolation: "branch"`) deprecated. |

## Implementation Plan

### Phase 1: `.gitignore` + Tracking Fix

**Goal:** Planning artifacts are tracked in git. `.gitignore` reflects reality.

1. Update `.gitignore`:
   - Remove blanket `.sf/` ignore
   - Add explicit runtime-only ignores (see proposed list above)

2. Force-add existing planning artifacts on current branch:
   ```
   git add --force .sf/milestones/ .sf/PROJECT.md .sf/DECISIONS.md .sf/REQUIREMENTS.md .sf/QUEUE.md
   ```

3. Ensure runtime files are NOT tracked:
   ```
   git rm --cached -r .sf/runtime/ .sf/activity/ .sf/STATE.md .sf/metrics.json .sf/completed-units.json .sf/auto.lock
   ```

4. Update README suggested `.gitignore` section

5. Remove `smartStage()` force-add of `SF_DURABLE_PATHS` — no longer needed since `.gitignore` doesn't block them

**Verification:** `git status` shows planning artifacts tracked, runtime files untracked. `git worktree add` on a new worktree has correct `.sf/milestones/` state.

### Phase 2: Remove Slice Branch Creation + Switching

**Goal:** No code creates, switches to, or references slice branches for new work.

1. Remove `ensureSliceBranch()` from `git-service.ts` (lines 485-544)
2. Remove `switchToMain()` from `git-service.ts` (lines 549-563)
3. Remove `getSliceBranchName()` from `worktree.ts` (lines 94-98)
4. Remove `isOnSliceBranch()` and `getActiveSliceBranch()` from `worktree.ts`
5. Update `auto.ts` dispatch paths — remove branch creation before `execute-task`
6. Update `handleAgentEnd` — remove branch-switching logic post-dispatch

**Verification:** Auto-mode runs a full slice (research → plan → execute → complete) without creating any branches. All commits land on `milestone/<MID>`.

### Phase 3: Remove Slice Merge Code

**Goal:** All slice→milestone and slice→main merge code is deleted.

1. Remove `mergeSliceToMilestone()` from `auto-worktree.ts` (lines 253-350)
2. Remove `mergeSliceToMain()` from `git-service.ts` (lines 705-893)
3. Remove merge dispatch guards from `auto.ts` (lines 1635-1679)
4. Remove `fix-merge` dispatch unit type from `auto.ts`
5. Remove `buildPromptForFixMerge()` from `auto.ts`
6. Remove `withMergeHeal()` from `git-self-heal.ts` (lines 99-136)
7. Remove `abortAndReset()` from `git-self-heal.ts` (lines 37-84) — or simplify to crash-recovery-only
8. Remove `shouldUseWorktreeIsolation()` preference resolution — worktree is the only mode
9. Remove `getMergeToMainMode()` — milestone merge is the only mode
10. Deprecate `git.isolation: "branch"` and `git.merge_to_main: "slice"` preferences

**Verification:** `git grep mergeSliceToMilestone` returns zero results. `git grep mergeSliceToMain` returns zero results. `git grep fix-merge` returns zero results (outside of changelog/docs).

### Phase 4: Simplify `mergeMilestoneToMain()`

**Goal:** Milestone→main merge is clean and minimal.

The function becomes:
1. Auto-commit any dirty state in worktree
2. `process.chdir(originalBasePath)` — back to main repo
3. `git checkout main`
4. `git merge --squash milestone/<MID>`
5. Build commit message with milestone summary + slice manifest
6. `git commit`
7. Optional: `git push`
8. `removeWorktree()` + `git branch -D milestone/<MID>`

No conflict categorization. No runtime file stripping (runtime files are gitignored, not in the merge). No `.sf/` special handling.

If squash-merge conflicts (parallel milestone edge case): stop auto-mode with clear error, user resolves manually or SF dispatches a one-time resolution session.

**Verification:** Complete a full milestone in auto-mode. `main` receives one squash commit with all code and planning artifacts.

### Phase 5: Test Cleanup

**Goal:** Test suite reflects the simplified architecture.

1. Delete or rewrite:
   - `auto-worktree-merge.test.ts` — tests slice→milestone merge (deleted)
   - `auto-worktree-milestone-merge.test.ts` — rewrite for simplified milestone→main
   - `worktree-e2e.test.ts` — rewrite for branchless flow
   - `worktree-integration.test.ts` — rewrite for branchless flow
   - Merge-related test cases in `git-service.test.ts`

2. Add new tests:
   - Branchless worktree lifecycle: create → commit → commit → squash-merge → cleanup
   - `.sf/` tracking: planning artifacts tracked, runtime files ignored
   - Manual worktree: `git worktree add` has correct `.sf/` state
   - Crash recovery: dirty state on milestone branch, restart, auto-commit, continue

3. Remove merge-specific doctor checks or simplify:
   - `corrupt_merge_state` — keep (still relevant for milestone→main)
   - `orphaned_auto_worktree` — keep
   - `stale_milestone_branch` — keep
   - `tracked_runtime_files` — keep

**Verification:** `npm run test` passes. No test references `mergeSliceToMilestone`, `mergeSliceToMain`, or `ensureSliceBranch`.

### Phase 6: Migration + Backwards Compatibility

**Goal:** Existing projects with slice branches continue to work.

1. State derivation (`deriveState()`) continues to read `sf/M001/S01` branch naming for legacy detection
2. On first run after upgrade:
   - Detect existing slice branches
   - Notify user: "SF no longer creates slice branches. Existing branches are preserved but new work commits directly to the milestone branch."
   - No forced migration — legacy branches are read-only context
3. Doctor check: `legacy_slice_branches` — informational, not auto-fix
4. Update `shouldUseWorktreeIsolation()` preference handling:
   - `git.isolation: "worktree"` → default behavior (only option)
   - `git.isolation: "branch"` → warning, treated as worktree
   - Remove preference UI for isolation mode

**Verification:** Open a project with existing `sf/M001/S01` branches. SF reads state correctly, new work commits on milestone branch without slice branches.

## Stress Test Results

Validated by three independent models:

### Gemini 2.5 Pro — 6 Attack Vectors

| Attack | Severity | Mitigation |
|--------|----------|------------|
| Parallel milestone code conflict at squash-merge | Medium | `git rebase main` before squash. Rare in single-user. |
| SQLite desync after `git reset --hard` | Low | DB rebuilt from tracked markdown on startup (M001/S02 importers). |
| Ghost lock after SIGKILL | Low | Existing heartbeat lock detection handles this. |
| Squash merge loses bisect granularity | Low | Commit messages tag slices. Branch preservable if needed. |
| Disk space with multiple worktrees | Low | Single active milestone at a time. Immediate cleanup. |
| Plan-action atomicity gap (crash between write and commit) | Low | `handleAgentEnd` auto-commits. Sequential model simplifies recovery. |

### GPT-5.4 (Codex) — Codebase-Informed Analysis

- Confirmed `smartStage()` force-add already implements tracked-artifact intent
- Confirmed `resolveMainWorktreeRoot` (PR #487) contradicts this architecture
- Confirmed `.sf/milestones/` partially tracked on `main` despite `.gitignore`
- Verdict: **Model is sound. Removes only accidental complexity.**

### GPT-5.4 (Codex) — Dissenting Opinion

Codex agreed on tracked artifacts and worktree-per-milestone, but pushed back on removing slice branches, calling it "a redesign, not a simplification." Specific concerns:

| Concern | Rebuttal |
|---------|----------|
| Crash recovery for orphaned slice branches disappears | The failure mode (orphaned branch needing merge) is caused by slice branches. Removing branches removes the failure. Sequential commits on one branch need no orphan recovery. |
| Concurrent edits to shared root docs (DECISIONS.md) from two terminals | Standard content conflict at squash-merge time. Not caused by or solved by slice branches. |
| Continuous integration via slice→milestone merges | In sequential single-user work, there's nothing to integrate against within the worktree. Pre-flight rebase before squash-merge is more direct. |
| Need a replacement slice-boundary primitive | Accepted: conventional commit tags (`feat(M001/S01):`) + optional git tags (`sf/M001/S01-complete`) serve as boundaries. |

Codex's analysis confirms the tracked-artifact approach but recommends treating branchless as a deliberate redesign with explicit replacement primitives, not a casual deletion.

### Edge Case: Two Milestones Touching Same Source Files

Scenario: M001 and M002 both modify `src/auth.ts`. M001 squash-merges first.

Resolution: Before M002 squash-merges, rebase onto updated `main`:
```
cd .sf/worktrees/M002
git fetch origin main
git rebase main
# Resolve any conflicts (code-only, never .sf/)
# Then squash-merge
```

This is standard git workflow. SF can automate the rebase step as a pre-merge check.

### Edge Case: Agent Crash Mid-Commit

Scenario: Power loss during `git commit` on the milestone branch.

Resolution: Git's internal journaling protects the object store. On restart:
- If commit completed: state is consistent
- If commit didn't complete: working directory has uncommitted changes, `handleAgentEnd` auto-commits on next dispatch
- No branch to be "stuck between" — single branch means no split-brain state

### Edge Case: User Edits Main While Worktree Active

Scenario: User makes manual commits on `main` while M001 worktree is active.

Resolution: Worktree is on `milestone/M001` branch, independent of `main`. Manual `main` commits don't affect the worktree. At squash-merge time, `git merge --squash` handles the divergence normally. If there's a conflict, it's resolved once.

## Metrics

### Before (Current)

| Metric | Value |
|--------|-------|
| Merge/conflict/branch code | 770+ lines across 4 files |
| Merge-related test files | 11 files |
| Branch types | 4 (main, milestone/*, sf/*/*, worktree/*) |
| Merge strategies | 3 (--no-ff, --squash, conflict resolution) |
| Dispatch unit types with merge logic | 2 (complete-slice, fix-merge) |
| Isolation modes | 2 (branch, worktree) |
| Doctor git checks | 4 |

### After (Proposed)

| Metric | Value |
|--------|-------|
| Merge/conflict/branch code | ~50 lines (simplified `mergeMilestoneToMain` only) |
| Merge-related test files | 3-4 files (rewritten) |
| Branch types | 2 (main, milestone/*) |
| Merge strategies | 1 (--squash) |
| Dispatch unit types with merge logic | 0 |
| Isolation modes | 1 (worktree) |
| Doctor git checks | 3-4 (simplified) |

### Net Impact

- **~720 lines deleted** (net, after simplified replacements)
- **~7 test files deleted or consolidated**
- **2 branch types eliminated**
- **2 merge strategies eliminated**
- **1 dispatch unit type eliminated** (fix-merge)
- **1 isolation mode eliminated** (branch)
- **0 merge conflicts possible within a worktree**

## Dependencies

- **M001 (Memory Database):** The SQLite database (`sf.db`) must remain gitignored. The M001/S02 importer layer rebuilds it from tracked markdown. This PRD's `.gitignore` update explicitly ignores `sf.db`.

- **PR #487:** Must be closed. The `resolveMainWorktreeRoot` approach (sharing `.sf/` across worktrees) contradicts tracked-artifact architecture.

## Open Questions

1. **Squash vs `--no-ff` for milestone→main merge?** Squash gives clean history on `main` but loses bisect granularity. `--no-ff` preserves granular commits but clutters `main`. Current proposal: squash (matching existing behavior), with option to preserve milestone branch for debugging.

2. **Should `worktrees/` move outside `.sf/`?** Having worktrees inside `.sf/` creates a nesting-doll pattern (worktree contains `.sf/` which is inside `.sf/worktrees/`). Relocating to `.sf-worktrees/` or `~/.sf/worktrees/<repo-hash>/` is cleaner but changes the filesystem layout. Recommendation: defer, address separately if it causes issues.

3. **Pre-flight rebase automation?** Before milestone→main squash-merge, should SF automatically `git rebase main`? Gemini recommends yes. Risk: rebase can fail with conflicts, adding a code path. Recommendation: implement as a doctor check ("milestone branch is behind main by N commits") with manual resolution, automate later if needed.