singularity-forge/docs/dev/drafts/tool-lazy-load-design.md

# SF Lazy Tool Surface Design Draft

Status: draft. Scope: design only.

## Prior Art

Claude Code and the local connector/tooling pattern use a small always-available discovery tool and defer detailed tool schemas until the model asks for a relevant group. This repo already has a nearby pattern in `src/resources/extensions/mcp-client/index.js:8-11`: `mcp_servers` lists configured servers, `mcp_discover` lazy-connects and registers discovered tools, and `mcp_call` remains a fallback call path. The design below applies that deferred-tool idea to SF-owned tools.

## Meta-Tool Schema

Tool name: `sf_tool_search`.

Input:

```ts
type SfToolSearchInput = {
  query: string;
  domain?: "planning" | "execution" | "memory" | "subagent" | "evidence" | "mcp" | "all";
  includeLoaded?: boolean;
  limit?: number;
};
```

Return:

```ts
type SfToolSearchResult = {
  query: string;
  loadedBefore: string[];
  matches: Array<{
    name: string;
    canonicalName: string;
    status: "loaded" | "available_lazy" | "deprecated_alias";
    title: string;
    description: string;
    domain: string;
    aliases?: string[];
    risk?: "read" | "write" | "destructive" | "external";
    schemaPreview: unknown;
    loadToken: string;
  }>;
  loadedAfter: string[];
  nextInstruction: string;
};
```

Behavior:

- `sf_tool_search` is always loaded.
- It searches a manifest-backed registry of lazy tools by name, alias, description, domain, and prompt snippet.
- For matched tools, it marks them loaded for the current session and causes the next LLM turn to include their full schemas.
- It should not execute the discovered tool. It only exposes schema and usage metadata.
- For tools with dangerous semantics, it returns risk metadata but does not bypass existing permission policy.

## Manifest Metadata

The current `extension-manifest.json` has a flat `provides.tools` string list. Lazy loading needs a richer backward-compatible form:

```json
{
  "provides": {
    "tools": [
      "complete_task",
      {
        "name": "memory_graph",
        "lazy": true,
        "domain": "memory",
        "aliases": [],
        "canonicalName": "memory_graph",
        "promptSnippet": "Inspect related SF memory records"
      }
    ]
  }
}
```

Prompt-build rule:

- Eager tools are passed to the provider normally.
- Lazy tools are omitted from the provider tool array and the Available tools prompt section.
- `sf_tool_search` gets a compact index of lazy names/domains/descriptions, not full schemas.
- When a lazy tool is loaded, its full schema and prompt guidelines are included from the next turn onward.

Suggested eager baseline:

- File/shell core: `read`, `edit`, `write`, `grep`, `glob`/`find` compatibility, `bash`, `ls`, `lsp`.
- Mandatory SF flow: `checkpoint`, `milestone_status`, `save_summary`, `plan_milestone`/future `milestone`, `plan_slice`/future `slice`, `plan_task`/future `task`, `complete_task`, `complete_slice`, `complete_milestone`, `validate_milestone`, `reassess_roadmap`.
- Delegation entrypoint: `subagent`.
- Meta discovery: `sf_tool_search`.

Suggested lazy groups:

- Subagent follow-up: `await_subagent`, `cancel_subagent`, `read_subagent`, `write_subagent`.
- Memory/evidence: `memory_search`, `memory_graph`, `query_journal`, `search_evidence`, `sift_search`, `codebase_search`.
- Admin/runtime: `resume_agent`, `kill_agent`, `read_output`.
- Issue/gate/chapter helpers: `report_issue`, `resolve_issue`, `record_gate`, `chapter_open`, `chapter_close`, `context_board`, `manage_todos`, `audit_product`.
- MCP: keep `mcp_servers` / `mcp_discover` / `mcp_call` under the existing MCP lazy-connect pattern.

## Session State

Loaded state belongs in the coding-agent session, not `.sf/sf.db`.

Shape:

```ts
type LazyToolSessionState = {
  loadedLazyTools: string[];
  loadedAtTurn: Record<string, number>;
  searchHistory: Array<{ turn: number; query: string; loaded: string[] }>;
};
```

Persistence:

- In-memory state is enough for a single active session.
- If session resume should preserve loaded lazy tools, serialize into the existing session transcript/state store alongside active tool names, not project DB.
- Do not write loaded-tool state to `.sf/sf.db`; this is provider/session prompt state, not project planning state.

Turn lifecycle:

1. User/agent starts with eager tool set plus `sf_tool_search`.
2. Model calls `sf_tool_search({ query })`.
3. Tool resolves matching lazy tools and updates `loadedLazyTools`.
4. Session calls `setActiveToolsByName([...eager, ...loadedLazyTools])`.
5. System prompt and provider tool list are rebuilt for the next LLM call.
6. Loaded tools stay loaded until session end, explicit unload, or compaction policy resets them.

## Current Architecture Feasibility

Partially achievable today, but full lazy prompt-build requires upstream coding-agent changes.

Current support:

- `packages/coding-agent/src/core/agent-session.ts:956-974` can set active tools by name and rebuild the system prompt for the next turn.
- `packages/coding-agent/src/core/extensions/loader.ts:534-548` can register/unregister extension tools and refresh the runtime registry.
- `packages/coding-agent/src/core/system-prompt.ts:157-178` already builds the visible tool list from selected tool names.
- MCP discovery already lazy-connects and can register external tools after discovery.

Missing pieces:

- Tool metadata has no first-class `lazy`, `aliases`, `canonicalName`, or `domain` fields in the active manifest flow.
- `extension-manifest.json` is a string list, so prompt-build cannot know which schemas to suppress.
- `sf_tool_search` needs a safe way to mutate active tools from inside a tool call or queue the mutation for the next turn.
- Prompt guidelines are appended for all registered tools today; lazy tools must not leak full usage guidance before loading.
- Provider telemetry should preserve old name, canonical name, and alias name for deprecation accounting.

Conclusion: a minimal SF-only prototype can register all tools but hide lazy ones from `setActiveToolsByName` until `sf_tool_search` loads them. A production implementation needs coding-agent metadata support so lazy schemas, prompt snippets, aliases, and deprecation state are handled consistently across SF, MCP, web/RPC, and provider compatibility code.