singularity-forge/docs/dev/ADR-015-flight-recorder.md

# ADR-015: Flight recorder via `charmbracelet/x/vcr`

**Date**: 2026-04-29
**Status**: proposed (deferred — capture for staged execution)

## Context

sf today writes:

- `.sf/event-log.jsonl` — structured event stream (phase changes, tool calls, errors).
- `.sf/traces/*.jsonl` — per-unit trace spans.
- `.sf/audit/` — historical state snapshots.

These are all *structured event streams*. They're great for programmatic analysis but they don't record what the auto-loop *looked like* on the operator's terminal — the actual TUI frames, the stream of tool output, the agent's thinking, the live progress indicators.

When something goes wrong in production (the auto-loop appears to hang, an agent generates surprising output, a hook misbehaves), the operator wants to **replay the session** — see what was on screen at minute 14 — not reconstruct it from JSON.

`charmbracelet/x/vcr` records terminal output as a sequence of frames and replays them deterministically. It's the right substrate for a flight recorder.

## Decision

- **Language: Go.** Standalone service or library; integrates with sf via shared filesystem (writes recordings to `.sf/recordings/`).
- **Recording substrate: `charmbracelet/x/vcr`** — captures ANSI/VT frames into a portable file format with timestamps.
- **Trigger: every auto-loop unit dispatch records by default.** Recording is opt-out per project via `.sf/config.toml` (`[telemetry] flight_recorder = false`).
- **Storage: `.sf/recordings/{unit-id}.vcr`**, with a retention policy (default 30 days, configurable). Old recordings auto-expire on the next sweep.
- **Replay: `sf replay <unit-id>`** — opens the recording in a TUI player; supports pause, scrub, frame-step, search-by-text.
- **Format: vcr-native.** No reinventing.

## Alternatives Considered

- **`asciinema`** — well-known terminal recorder, mature tooling, JSON-based format.
  - *Rejected:* asciinema runs as a subprocess wrapping the shell. Integrating with sf's auto-loop (which is the *driver*, not a child of the recorder) requires inverting the model. `vcr` is library-shaped — sf calls into it.
- **`vhs`** — Charm's CLI video recorder, used for demos.
  - *Rejected:* `vhs` is for scripted demos, not live capture. Wrong tool.
- **Re-render from `.sf/event-log.jsonl`** — replay events through pi-tui to reproduce the frames.
  - *Rejected:* requires keeping pi-tui forever, and rendering depends on terminal geometry that may differ from the original. Frame-accurate replay is not the same as event replay; both have value but they're different products.
- **Build a custom recorder.**
  - *Rejected:* `vcr` exists. NIH-don't.

## Consequences

**Positive**

- **Frame-accurate post-mortem** — when a unit fails or the auto-loop hangs, the operator sees exactly what was on screen, including timing.
- **Onboarding artefact** — recordings of "what does sf do?" become shareable demos without scripting.
- **Audit trail for destructive ops** — admin actions in the future Charm TUI client (ADR-017) and Singularity Memory admin UI (ADR-014) can be recorded for security audit.
- **Light coupling** — `vcr` is a Go library; sf's TS core invokes a small Go recorder process per unit dispatch. No tight integration with the agent loop.

**Negative**

- **Disk usage** — recordings are bigger than event logs (frame data vs. structured records). Mitigated by retention policy. Estimate: ~1MB per 10-minute unit at typical TUI density.
- **Operator-only** — frame replay isn't useful in headless contexts. Headless dispatches should disable recording (`SF_FLIGHT_RECORDER=0` env).
- **Polyglot crosses one more boundary** — sf core (TS) writes recordings via a Go subprocess. Same shape as ADR-013 (TS↔Go via stdio); manageable.

**Risks and mitigations**

- *Risk:* `vcr` API churn — it's in `charmbracelet/x` (experimental).
  - *Mitigation:* pin a version; abstract recording behind an interface so a future swap is contained.
- *Risk:* Recording overhead measurably slows the auto-loop.
  - *Mitigation:* benchmark before enabling-by-default. If overhead > 5%, ship as opt-in only.
- *Risk:* Sensitive data (tokens, paths, secrets) leaks into recordings.
  - *Mitigation:* same redaction layer as `event-log.jsonl` — applied at the frame level before write. Enforce via a redaction filter applied to the VT stream.

## Out of Scope

- **Audio recording.** Terminal frames only.
- **Cross-host recording** — each host records its own units; flight-recorder doesn't try to stitch SSH-worker output onto orchestrator-side replay. (Each unit attempt has a `worker_host`; replay is per-host.)
- **Live remote viewing** of an in-progress recording — that's a different feature (could be Wish + Bubble Tea showing a "live" view of the auto-loop). Track separately if wanted.

## Sequencing

| When | Action |
|---|---|
| Tier 2/3 — after federation primitives land | Build a thin Go recorder process; sf core spawns one per unit dispatch. |
| Tier 3 | `sf replay <unit-id>` command — TUI player using Bubble Tea. |
| Tier 3 | Redaction filter parity with `event-log.jsonl`. |
| Tier 4 (nice-to-have) | Retention policy auto-sweep; recording bundle export (`sf recording export <unit-id>` → `.vcr.tar.gz` for sharing). |

## Out of Scope (continued — feature-creep guardrails)

- AI-assisted summarisation of recordings ("show me what failed in the last 5 unit attempts") — possible later via fantasy + recording metadata, but explicitly not v1.
- Web-based replay UI — server-rendered replay is a separate product surface; v1 is local TUI only.

## References

- `charmbracelet/x/vcr` — terminal recording library.
- `SPEC.md` §19 — Observability (where structured event logs and traces live).
- `ADR-016` — Charm AI stack adoption (frames why Go for new services).
- `ADR-017` — Charm TUI client (future replay UI consumer).