---
name: systematic-debugging
description: Use for any bug, failing test, unexpected behaviour, or production regression in sf. Evidence first — logs, traces, event log, code search — before proposing any fix. Side-chain skill that returns to the caller after the fix is verified.
---

# Systematic Debugging

## Iron Law

```
NO FIX WITHOUT ROOT-CAUSE INVESTIGATION FIRST.
NO NON-TRIVIAL FIX WITHOUT AN EXPLICIT REPRO.
```

If you haven't completed Phase 1, you cannot propose fixes.

## Recognize Your Own Rationalizations

Debugging is where the LLM's verification failure modes hit hardest. You will feel the urge to skip evidence collection and jump to a fix. These are the exact excuses you reach for — recognize and invert each:

- "I bet I know what this is — let me just try the fix." → No. The fix you have in mind is a hypothesis. Phase 1 (Evidence) and Phase 2 (Root Cause) are not optional.
- "The error message obviously says it's X." → Error messages name the symptom. They rarely name the root cause. Read the stack trace; find the line; verify the call site.
- "The test was probably flaky — let me re-run it." → Re-running a test that failed is not a fix; it is hoping the bug went away. If a re-run greens, the failure was not flaky — it was timing-dependent or order-dependent, which is a real bug.
- "The diff is small, so the change can't be the cause." → Small diffs ship the most subtle regressions. Confirm via `git log` / `git bisect`, not by ruling out by size.
- "It works on my machine." → "Your machine" is one execution. If the failure is reproducible, reproduce it; if not, capture the environment difference *as a finding*, not as a dismissal.

If you catch yourself proposing a fix before Phase 1 evidence is in writing — stop. Read the logs, the trace, the event log. Then decide.

## When to Run

- A test fails, RED happens for the wrong reason, GREEN regresses, build fails, lint fails.
- An sf autonomous-loop unit hits an unexpected error, abandons, or stalls.
- Production behaviour does not match the contract.
- Reviewer reports a real bug.

## Skill Chain

Side-chain skill. Interrupts the main delivery chain, fixes the issue, returns.

```
← prev:  any skill in the main chain (invoked on bug/failure)
→ next:  return to the interrupted skill (auto-continues after fix verified)
```

Debugging is part of delivery, not a separate approval gate. Do not pause for permission; resume the main chain after the fix is verified.

## Phase 1 — Gather Evidence

Smallest reliable reproduction first. Be specific about what fails, where it fails, and what changed recently.

```bash
# Code/git context
rg -n "<symbol>|<error message>|<route>" src/ packages/
git log -- <path>
git diff -- <path>

# sf runtime evidence
ls -lat .sf/event-log.jsonl .sf/traces/ 2>/dev/null
tail -200 .sf/event-log.jsonl | jq 'select(.kind=="error" or .level=="error")'
ls .sf/active/                                  # in-flight units
ls .sf/audit/                                   # historical audit dumps
```

For LLM/provider/transport bugs, capture all of:

- canonical `provider/model` requested
- which provider transport handled the call
- whether the failure was pre-resolution, in-transport, or inside the provider
- whether the trace can reproduce it

For native-engine bugs (`forge_engine.node`):

```bash
ldd rust-engine/npm/linux-x64-gnu/forge_engine.node 2>&1 | grep -E "not found|missing"
```

When evidence collection splits into independent streams, fan out with parallel subagents (`Explore` for repo search, `Plan` for design analysis). Keep the root-cause + repro doctrine in this skill.

## Phase 2 — Identify Root Cause

1. Read error messages completely — stack traces, file paths, line numbers.
2. Recent changes: `git log --since="2 days ago"` — what shipped before this broke?
3. Find a working example: `rg` for similar usages that work.
4. Compare working vs broken — list every difference.
5. State the hypothesis: "I think X is the root cause because Y."

Label all claims:

- `Observed:` directly from logs, diffs, tests, traces.
- `Inferred:` explanation supported by those facts.
- `Proposed:` fix or next experiment.
- `Confidence:` 0.0–1.0 with one-line reason.
- `Falsifier:` what result would disprove this diagnosis.
- `Reflection:` weakest assumption, next diagnostic check.

### Confidence Gate (Phase 2 → Phase 3)

| Gate | Threshold | Below threshold |
|---|---|---|
| Root cause identified | 0.90 | Run `advisory-partner` adversarial wave (advocate + challenger). |
| Fix verified (Phase 4) | 0.95 | Re-run repro path. If still below, expand the wave. |

Skip the gate for trivial failures (typo, missing import, obvious one-line stack trace).

## Phase 3 — Test Hypothesis Minimally

**Bug = missing correct-behaviour test.**

1. Write the failing test specifying the correct behaviour. It must fail (RED) because the bug exists.
2. If it passes immediately → the test is testing the broken behaviour. Fix the test first.
3. Run via `npm run test:unit -- <pattern>` or the relevant compiled test target.
4. If hypothesis validated → Phase 4. If not → new hypothesis, back to Phase 2.

Use `spec-first-tdd` for the contract test if the bug crosses a non-trivial boundary.

## Phase 4 — Fix and Verify

1. Fix the **root cause**, not the symptom.
2. `npm run typecheck:extensions` — clean.
3. `npm run test:unit -- <pattern>` — regression test green.
4. For non-trivial fixes: re-run the same repro/debug path; capture the **solved** boundary output.
5. For runtime fixes: tail the event log and confirm the error class disappears.

### Trace format for verification

Every claimed PASS in the slice artefact must be backed by a trace — paraphrase is not evidence:

```
### Check: <what was verified — repro path, regression test, etc.>
**Command run:** <exact command>
**Output observed:** <copy-paste of stdout/stderr — truncate if long, keep the relevant part>
**Result: PASS** (or FAIL — with Expected vs Actual)
```

A check without a `Command run` block is a skip. "I re-ran the repro and it works now" without the command and the output is a self-report, not verification.

Persist the pattern to memory so future units don't re-hit it:

```
log_reasoning(
  category="anti-pattern",
  content="<symptom> in <component> — root cause: <one line> — fix: <approach> — test: <name>",
  confidence=0.9
)
```

## Subagent Patterns

Use only when evidence collection runs in parallel:

- `Explore` agent — map a failing code path quickly without burning the main context.
- `Plan` agent — design analysis when the failure crosses an architectural boundary.
- An adversarial reviewer via `advisory-partner` when diagnosis confidence is below threshold.

Do not delegate the fix — the main thread owns it. Subagents return paragraphs; you apply the change.

## Milestone Bug Registry Guidance

Some milestones (notably systematic bug-hunt milestones such as M008) ship with a curated `bug-registry.json` and a `triage-protocol.md` under `.sf/milestones/<MID>/`. When you're fixing a bug and either artifact is present for the active milestone, follow these additional rules:

1. **Read the registry first** — before proposing a fix, check `.sf/milestones/<MID>/bugs/bug-registry.json` to see if the issue is already documented. If it is, use the `suggestedFix` field as a starting hypothesis, not as gospel. (Canonical example: `.sf/milestones/M008/bugs/bug-registry.json`.)
2. **Follow the triage protocol** — classify severity using the definitions in `.sf/milestones/<MID>/triage-protocol.md`. HIGH and MEDIUM severity bugs require regression tests; LOW severity bugs should be batched with other fixes in the same cluster.
3. **Update the registry** — after fixing, update `bug-registry.json`: set `status` to `"FIXED"`, add `fixedByTaskId`, and remove `assignedTo` if present.
4. **Cluster-aware fixes** — if a fix touches files in multiple clusters, escalate to human review per the triage protocol escalation rules.
5. **Confidence gate for batch fixes** — when fixing multiple LOW-severity bugs in one PR, each item must have a passing regression test and the batch confidence must be ≥ 0.80.

When the active milestone has no `bug-registry.json`, this section does not apply — proceed with the normal evidence/repro/fix flow above.

### Registry Fields Reference

| Field | Meaning |
|---|---|
| `id` | Unique finding identifier (`<cluster>-<file>-<description>-<line>`) |
| `file` | Source file where issue was found |
| `lines` | Line number(s) |
| `category` | Bug category (e.g. `logic`, `race condition`, `resource leak`) |
| `severity` | `HIGH` / `MEDIUM` / `LOW` / `FALSE_POSITIVE` |
| `status` | `CONFIRMED` / `FALSE_POSITIVE` / `FIXED` / `WONTFIX` / `IN_PROGRESS` |
| `cluster` | Which cluster the finding belongs to |
| `suggestedFix` | Recommended fix or `"N/A (intentional design)"` |
| `fixedByTaskId` | Task ID that fixed the finding (set after fix) |

## Rules

- Fix root cause, not symptom.
- Write or update the regression test first.
- Keep before/after repro for non-trivial bugs.
- Three failed fix attempts → stop. Question the architecture. Discuss before attempt 4.
- Never claim "fixed" without re-running the original repro path.
- **Registry-backed milestones (e.g., M008)**: Never claim "fixed" without updating `bug-registry.json`.

## Red Flags — STOP

- Proposing a fix before checking logs/traces/event log.
- Proposing a non-trivial fix from traces alone when repro is feasible.
- "Probably X, let me fix that" without verification.
- Diagnosis stated without `Observed:` evidence.
- Third fix attempt without questioning architecture.
- Test deleted to make a build green — that's encoding the bug, not fixing it.
- Fixing a bug listed in a milestone bug-registry without reading the registry first.
