diff --git a/docs/dev/MEMORY-SYSTEM-INTEGRATION-GUIDE.md b/docs/dev/MEMORY-SYSTEM-INTEGRATION-GUIDE.md new file mode 100644 index 000000000..478fcdd43 --- /dev/null +++ b/docs/dev/MEMORY-SYSTEM-INTEGRATION-GUIDE.md @@ -0,0 +1,284 @@ +# Memory System Integration Guide + +Quick-start guide for using SF's autonomous memory system in extensions and dispatch logic. + +## Quick Start: Record Unit Outcomes + +```typescript +import { recordUnitOutcomeInMemory } from './uok/unit-runtime.js'; + +// After a unit completes execution: +recordUnitOutcomeInMemory(unit, 'completed', { + success: true, + duration: 2341, + modelUsed: 'claude-sonnet-4-6' +}); + +// Fire-and-forget: +// - Stores pattern in memory (never blocks) +// - Success: 0.9 confidence (strong signal) +// - Failure: 0.5 confidence (weaker signal) +``` + +## Quick Start: Enhance Dispatch with Memory + +```typescript +import { enhanceUnitRankingWithMemory } from './auto-dispatch.js'; + +const candidates = [ + { id: 'unit-a', type: 'research', readiness: 0.8 }, + { id: 'unit-b', type: 'refactor', readiness: 0.6 } +]; + +const baseScores = { 'unit-a': 0.8, 'unit-b': 0.6 }; +const enhanced = await enhanceUnitRankingWithMemory(candidates, baseScores); + +// Returns: candidates with memory-boosted scores +// Max boost: 15% of pattern confidence (conservative) +// Result: unit-a boosted if similar patterns succeeded before +``` + +## Quick Start: Add Context to Gate Results + +```typescript +import { enrichGateResultWithMemory } from './uok/gate-runner.js'; + +// After gate evaluation: +const gateResult = { outcome: 'fail', reason: 'timeout' }; +const enriched = await enrichGateResultWithMemory(gateResult, 'deployment-gate'); + +// Returns: +// { +// outcome: 'fail', // Gate decision unchanged +// reason: 'timeout', +// memoryContext: { +// hasHistoricalPattern: true, +// similarFailures: [ +// { id: 'm1', confidence: 0.92, content: 'Network timeout...' } +// ] +// } +// } +``` + +--- + +## Core Memory Operations + +### 1. Create a Memory Entry + +```typescript +import { createMemory } from './memory-store.js'; + +const id = createMemory({ + category: 'gotcha', + content: 'TypeScript strict mode requires null checks on optional params', + confidence: 0.95 +}); +``` + +**Categories:** +- `gotcha` — Known issues, workarounds, edge cases +- `convention` — Coding patterns, naming standards +- `architecture` — Design decisions, responsibilities +- `pattern` — Recurring execution patterns +- `environment` — Config, setup, deployment +- `preference` — Optimization decisions + +**Confidence:** 0.0-1.0 (default 0.8) + +### 2. Query Memory by Similarity + +```typescript +import { getRelevantMemoriesRanked } from './memory-store.js'; + +const embedding = computeUnitEmbedding('research-type'); +const memories = await getRelevantMemoriesRanked( + embedding, + 'pattern', // Category filter + 5 // Top 5 results +); + +// Returns sorted by cosine similarity + confidence +``` + +### 3. Create Relationships + +```typescript +import { createRelation } from './memory-relations.js'; + +createRelation( + failureMemoryId, + rootCauseMemoryId, + 'caused_by', + 0.9 +); +``` + +**Relation Types:** +- `caused_by` — Unit failure → root cause +- `similar_to` — Pattern similarity +- `workaround_for` — Known fix +- `depends_on` — Architectural dependency + +### 4. Query Relationships + +```typescript +import { traverseGraph } from './memory-relations.js'; + +const graph = traverseGraph(startMemoryId, maxDepth); +// Returns: { nodes, edges } +``` + +--- + +## Common Integration Patterns + +### Pattern 1: Learn from Success/Failure + +```typescript +// In unit-runtime or dispatch logic: +if (unitCompleted) { + const pattern = { + unitType: unit.type, + outcome: unit.status, + modelUsed: selectedModel, + duration: executionTime, + toolsNeeded: requiredTools + }; + + recordUnitOutcomeInMemory(unit, unit.status, pattern); +} +``` + +### Pattern 2: Boost Dispatch Based on History + +```typescript +// In dispatch decision logic: +const baseScores = computeBaseScores(candidates); +const enhanced = await enhanceUnitRankingWithMemory(candidates, baseScores); +const selected = enhanced.reduce((best, curr) => + curr.score > best.score ? curr : best +); +``` + +### Pattern 3: Diagnostic Context on Failures + +```typescript +// In gate evaluation: +const gateResults = await evaluateAllGates(unit); +const enriched = await Promise.all( + gateResults.map(result => + enrichGateResultWithMemory(result, result.gateId) + ) +); +// Enriched results now have historical context +``` + +### Pattern 4: Manual Memory Capture + +```typescript +// From SF CLI or tools: +import { executeMemoryCapture } from './tools/memory-tools.js'; + +executeMemoryCapture({ + category: 'convention', + content: 'Always validate input before processing', + confidence: 0.85 +}); +``` + +--- + +## Graceful Degradation + +All memory operations degrade gracefully: + +```typescript +// If DB unavailable → operation fails silently +// If memory lookup fails → continue without boost +// If embedding computation fails → use default + +// Example: safe dispatch enhance +try { + const enhanced = await enhanceUnitRankingWithMemory(units, scores); + return enhanced; // Use memory-boosted scores +} catch (err) { + console.warn('Memory enhance failed, using base scores'); + return units.map((u, i) => ({ ...u, score: scores[u.id] || 0 })); +} +``` + +--- + +## Performance Notes + +| Operation | Latency | Notes | +|-----------|---------|-------| +| `createMemory()` | <5ms | Fire-and-forget async | +| `getRelevantMemoriesRanked()` | 10-50ms | Depends on DB size | +| `cosineSimilarity()` | <1ms | 128D vectors | +| `computeEmbedding()` | 5-20ms | Deterministic hash | +| Dispatch boost overhead | <10ms | Per dispatch cycle | + +**Best Practices:** +- Never block on memory operations +- Use fire-and-forget async for recording +- Cache embeddings if used repeatedly +- Batch memory queries when possible + +--- + +## Testing with Memory + +```typescript +// Mock memory in tests: +vi.mock('./memory-store.js', () => ({ + getRelevantMemoriesRanked: vi.fn().mockResolvedValue([ + { id: 'm1', confidence: 0.9, content: 'test memory' } + ]), + createMemory: vi.fn().mockReturnValue('m-new-id'), + isDbAvailable: vi.fn().mockReturnValue(true) +})); + +// Test memory-enhanced dispatch: +it('boosts matching unit scores', async () => { + const enhanced = await enhanceUnitRankingWithMemory(candidates, scores); + expect(enhanced[0].score).toBeGreaterThan(scores[candidates[0].id]); +}); +``` + +--- + +## Debugging Memory + +```typescript +import { getActiveMemoriesRanked } from './memory-store.js'; + +// List all memories: +const all = getActiveMemoriesRanked(1000); + +// Filter by category: +const gotchas = all.filter(m => m.category === 'gotcha'); + +// Check hit counts: +all.forEach(m => console.log(`${m.id}: ${m.hit_count} hits`)); +``` + +--- + +## Architecture: Memory is Internal + +- **No MCP server** — memory stays inside SF +- **SQLite only** — Node 24 native (no external deps) +- **Fire-and-forget** — never blocks dispatch +- **Private learning** — autonomous pattern extraction + +--- + +## See Also + +- `docs/dev/MEMORY-SYSTEM-ARCHITECTURE.md` — Full architecture reference +- `src/resources/extensions/sf/uok/unit-runtime.js` — Unit recording +- `src/resources/extensions/sf/auto-dispatch.js` — Dispatch ranking +- `src/resources/extensions/sf/uok/gate-runner.js` — Gate enrichment +- `src/resources/extensions/sf/tools/memory-tools.js` — CLI access