singularity-forge/.plans/native-perf-optimizations.md

# Native Performance Optimizations — deriveState, JSONL, Paths, Parsing

## Overview

Four native Rust optimizations to eliminate hot-path bottlenecks in GSD's dispatch cycle.
Building on the existing git2 migration and native parser infrastructure.

---

## 1. Native deriveState — Eliminate Frontmatter Re-serialization

### Problem
`state.ts:134-176` — When `nativeBatchParseGsdFiles()` returns parsed files, the JS
side re-serializes frontmatter back into YAML strings so downstream parsers can re-parse
them. This is a round-trip waste: Rust parses → JS re-serializes → JS re-parses.

### Solution
The native batch parser already returns `{ metadata: JSON, body, sections }`.
Instead of re-serializing frontmatter to YAML in JS, modify `cachedLoadFile()` to
return the raw body directly, and update downstream parsers to accept pre-parsed
metadata. This eliminates the entire lines 143-172 re-serialization loop.

However, the parsers (`parseRoadmap`, `parseSummary`, `parsePlan`, etc.) all expect
raw markdown strings with frontmatter. Changing their signatures would be a massive
refactor. Instead:

**Approach: Make Rust return the original file content alongside parsed data.**

Add a new field `rawContent: String` to `ParsedGsdFile` that contains the complete
original file content. The JS batch cache stores this directly, eliminating the
re-serialization entirely. Downstream parsers get exactly what `loadFile()` would return.

### Implementation
- **Rust** (`gsd_parser.rs`): Add `raw_content` field to `ParsedGsdFile`, populate with
  the original file content read from disk.
- **TS** (`native-parser-bridge.ts`): Expose `rawContent` in `BatchParsedFile`.
- **TS** (`state.ts`): Replace the 30-line re-serialization loop with
  `fileContentCache.set(absPath, f.rawContent)`.

### Impact
Eliminates ~30 lines of JS string building per dispatch. Removes JSON.parse of metadata
that was only used to re-serialize back to YAML.

---

## 2. Native JSONL Streaming Parser

### Problem
`session-forensics.ts:68-78` — Parses JSONL by `split("\n").map(JSON.parse)` with a
10MB cap. Large session files cause OOM or slowness.

### Solution
Add a Rust JSONL parser that streams through the file with constant memory, returning
structured data. Uses `serde_json` for parsing and handles arbitrary file sizes.

### Implementation
- **Rust** (`gsd_parser.rs`): Add `parse_jsonl_tail(path, max_entries?)` function that:
  1. Memory-maps or streams the file from the tail
  2. Parses each line as JSON
  3. Returns the last N entries as a JSON array string
- **TS** (`native-parser-bridge.ts`): Add bridge function.
- **TS** (`session-forensics.ts`): Use native parser, fall back to JS implementation.

### Impact
Handles arbitrary file sizes. 3-5x faster parsing on 10MB files.

---

## 3. Native Directory Tree Index

### Problem
`paths.ts:20-34` — `cachedReaddirSync()` caches per-directory, but caches are
cleared every dispatch via `invalidateAllCaches()`. Each `resolveMilestoneFile`,
`resolveSliceFile`, `resolveTaskFile` triggers separate directory reads.

### Solution
Add a Rust function that walks the entire `.gsd/` tree once and returns a flat
file listing. The JS side builds a Map from this, making all path resolution O(1)
lookups instead of repeated `readdirSync` + regex matching.

### Implementation
- **Rust** (`gsd_parser.rs`): The `batchParseGsdFiles` already walks the tree.
  Add `scan_gsd_tree(directory)` that returns `Vec<{ path, isDir, name }>` for
  ALL entries (not just .md files).
- **TS** (`native-parser-bridge.ts`): Add bridge function.
- **TS** (`paths.ts`): Add native tree cache. On first access, call native scan
  and build lookup maps. `clearPathCache()` clears the native cache too.

### Impact
Eliminates 20-50 `readdirSync` calls per dispatch. Makes `resolveDir`/`resolveFile`
O(1) lookups.

---

## 4. Expand Native Markdown Parsing

### Problem
`files.ts` parsers (`parsePlan`, `parseSummary`, `parseContinue`) still use JS regex.
Each runs ~10-20 regex patterns per file. Only `parseRoadmap` has a native implementation.

### Solution
Add native Rust implementations for `parsePlan` and `parseSummary` — the two parsers
called most frequently during `deriveState`. `parseContinue` is called infrequently
and can stay in JS.

### Implementation
- **Rust** (`gsd_parser.rs`): Add `parse_plan_file(content)` and `parse_summary_file(content)`.
- **TS** (`native-parser-bridge.ts`): Add bridge functions with JS fallback.
- **TS** (`files.ts`): Call native versions first, fall back to JS.

### Impact
3-5x faster parsing per file. With ~20 files per deriveState, saves 20-40ms.

---

## Implementation Order

1. **deriveState raw content** (smallest change, biggest immediate impact)
2. **Directory tree index** (eliminates readdirSync overhead)
3. **JSONL streaming parser** (helps crash recovery path)
4. **Plan/Summary native parsers** (improves parsing throughput)

## Files Modified

### Rust
- `native/crates/engine/src/gsd_parser.rs` — new functions + rawContent field

### TypeScript
- `src/resources/extensions/gsd/native-parser-bridge.ts` — new bridge functions
- `src/resources/extensions/gsd/state.ts` — simplified batch cache
- `src/resources/extensions/gsd/paths.ts` — native tree cache
- `src/resources/extensions/gsd/session-forensics.ts` — native JSONL
- `src/resources/extensions/gsd/files.ts` — native plan/summary parsers