singularity-forge/src/loader.ts
Andrew 815be0a698 feat: managed RTK integration with opt-in preference and web UI toggle (#2620)
* feat: integrate managed RTK across shell workflows

* fix(rtk): unify managed fallback and live savings wiring

* fix(rtk): improve TUI status visibility

* fix(tests): make portability tests independent of pi-coding-agent dist build

The CI portability test runs don't guarantee that
packages/pi-coding-agent has been compiled. Any test that
imported files pulling in @gsd/pi-coding-agent (resource-loader,
preferences-skills, async-bash-tool, etc.) crashed with
ERR_MODULE_NOT_FOUND pointing at dist/index.js.

Two changes to dist-redirect.mjs (the Node ESM loader hook used by
all unit tests):
- Redirect the bare @gsd/pi-coding-agent specifier to the workspace
  source entrypoint (src/index.ts) so no dist/ artifact is needed.
- Extend the load() hook to transpile *.ts files under
  packages/pi-coding-agent/src/ through TypeScript's transpileModule.
  Node's --experimental-strip-types can't handle parameter properties
  and similar syntax present in that package's source; full transpilation
  avoids the ERR_UNSUPPORTED_TYPESCRIPT_SYNTAX crash.

Also fix the dashboard.tsx responsive grid:
- xl:grid-cols-5 → xl:grid-cols-4 2xl:grid-cols-5
  (5 metric cards no longer fit at xl without overflow; test contract
  expected xl:grid-cols-4)
- Keep loading-skeletons.tsx in sync with the same breakpoints.

Add src/tests/resolve-ts-loader.test.ts to guard the loader behaviour:
- bare @gsd/pi-coding-agent redirect points to workspace source
- direct source-entry rewrite (.js → .ts)
- transpilation removes TS parameter property syntax that strip-only
  mode cannot parse

* fix(tests): redirect all workspace package imports to source in portability tests

The previous fix only redirected @gsd/pi-coding-agent to its
source entrypoint. In CI, pi-coding-agent/src itself imports
@gsd/pi-ai (and other workspace packages) which were still pointing
at dist/. Since no workspace dist is built during the portability
test run, any transitive resolution hit the same ERR_MODULE_NOT_FOUND.

Changes to dist-redirect.mjs:
- Redirect @gsd/pi-ai, @gsd/pi-ai/oauth, @gsd/pi-agent-core, and
  @gsd/pi-tui bare imports to their workspace src/ entrypoints.
- Broaden the load() transpilation condition from
  '/packages/pi-coding-agent/src/' to '/packages/*/src/' so that
  all workspace source files are run through TypeScript's
  transpileModule, handling parameter properties and other syntax
  that Node's strip-only mode rejects.

Verified by hiding all four workspace dist/ directories locally and
running the failing test set — 96/96 pass.

* fix(tests): redirect @gsd/native sub-paths; fix Windows .cmd spawnSync

Two more portability failures after the previous fix:

1. @gsd/native sub-path imports (@gsd/native/fd, @gsd/native/text, etc.)
   were not redirected — the loader only handled the bare specifier.
   Added a prefix-match redirect for @gsd/native/* → packages/native/src/<sub>/index.ts.

2. Windows RTK tests failed because createFakeRtk produces a .cmd wrapper
   on Windows, and spawnSync(binaryPath, [...]) without shell:true silently
   returns non-zero when the binary is a .cmd file.
   Added shell: /\.(cmd|bat)$/i.test(binaryPath) to the spawnSync calls in:
   - src/resources/extensions/shared/rtk.ts (rewriteCommandWithRtk)
   - src/resources/extensions/shared/rtk-session-stats.ts (readCurrentRtkGainSummary)
   - packages/pi-coding-agent/src/utils/rtk.ts (rewriteCommandForGsd)
   Production use of rtk.exe is unaffected; the shell flag is only true for
   .cmd/.bat paths.

Verified: all 93 portability tests pass with all workspace dist/ directories
removed (simulating CI portability environment).

* fix(tests): Windows portability fixes — HOME env, managed RTK path, perf threshold

Four Windows-specific failures fixed:

1. app-smoke.test.ts: process.env.HOME is undefined on Windows (uses
   USERPROFILE instead). Changed to homedir() from node:os which works
   cross-platform.

2. Managed RTK path tests on Windows: tests placed a fake RTK as rtk.exe
   (by copying a .cmd script into a .exe filename), which Windows cannot
   execute. Two-part fix:
   - resolveRtkBinaryPath() in both rtk.ts files now falls back to rtk.cmd
     in the managed dir on Windows when rtk.exe is absent.
   - withManagedFakeRtk and equivalent patterns in rtk.test.ts,
     rtk-session-stats.test.ts, rtk-execution-seams.test.ts changed to
     place the fake at rtk.cmd instead of rtk.exe on Windows.

3. bg_shell RTK test on Windows: requires bash (for shell sessions), which
   is not available on the blacksmith-4vcpu-windows-2025 runner without
   Git Bash installed. Test now skips on win32.

4. derive-state-db perf assertion: 10ms threshold was too tight for Windows
   CI runners (measured 12ms under load). Raised to 25ms — still catches
   real regressions (baseline is 3ms locally and ~12ms on stressed runners).

* fix(tests): fix managed RTK path fallback on Windows in src/rtk.ts + fix copyable fake

Two remaining Windows failures:

1. src/rtk.ts was never patched with the rtk.cmd managed-dir fallback
   (only the shared/rtk.ts and pi-coding-agent/src/utils/rtk.ts were updated).
   Added the same rtk.cmd fallback and shell:.cmd detection to src/rtk.ts,
   which is what rtk.test.ts imports from.

2. createFakeRtk on Windows wrote '%~dp0\fake-rtk.js' in the .cmd content —
   this resolves relative to the .cmd file's own directory. When the test
   copies rtk.cmd to a different managed dir, %~dp0 resolves to the copy
   destination where fake-rtk.js does not exist. Fixed by embedding the
   absolute path to fake-rtk.js directly in the .cmd content so the fake
   works correctly regardless of where the .cmd is copied.

* feat(experimental): add RTK opt-in preference with web UI toggle

- Add `experimental` category to GSDPreferences with `rtk: boolean` (default: false)
- RTK is now opt-in: disabled by default for all projects unless explicitly enabled
- Validate experimental.* keys; unknown experimental keys produce warnings

Web UI:
- Add ExperimentalPanel component with animated toggle switch per flag
- Add /api/experimental route (GET/PATCH) to read/write flags in preferences.md
- Add 'Experimental' tab to settings dialog sidebar nav (FlaskConical icon)
- Include ExperimentalPanel at bottom of gsd-prefs mega-scroll
- Fix toggle disabled state: trigger loadSettingsData for 'experimental' section
  and self-fetch on mount when data is absent

Dashboard:
- Gate RTK Saved metric card on rtkEnabled from live auto state (web)
- Gate TUI dashboard RTK savings row on rtkEnabled
- Gate TUI footer RTK status updates on experimental.rtk preference
- Propagate rtkEnabled through AutoDashboardData → bridge-service → store

Build:
- Add scripts/build-if-stale.cjs: incremental build driver that skips each
  step (packages, root tsc, copy-resources, web) when output is newer than
  source; replaces full rebuild chain in gsd:web
- Add scripts/web-stop.cjs: robust stop with registry + legacy PID + orphan
  sweep via pgrep; handles crash/restart orphaned next-server processes
- gsd:web now uses build-if-stale.cjs (fast cold starts, instant when unchanged)
- gsd:web:stop / gsd:web:stop:all use web-stop.cjs directly

Fix: correct import path in rtk-status.ts (./preferences.js not ../preferences.js)

* fix: restore em-dash encoding in package.json to match upstream

* refactor(rtk): move command rewrite out of pi-coding-agent into GSD extension

Per review feedback from igouss: pi-coding-agent should not be modified to add
GSD-specific logic. Instead, add a proper extension point and wire RTK through it.

Changes to packages/pi-coding-agent (extension API only — no RTK logic):
- Add BashTransformEvent + BashTransformEventResult types to extension API
- Add on('bash_transform') overload to ExtensionAPI interface
- Add emitBashTransform() to ExtensionRunner (chains all handlers in order)
- Call emitBashTransform() in wrapToolWithExtensions before bash tool execution
- Export new types from extensions/index.ts and package index.ts
- Revert all RTK-specific changes from bash-executor.ts, tools/bash.ts
- Remove packages/pi-coding-agent/src/utils/rtk.ts entirely

Changes to GSD extension:
- Register bash_transform handler in register-hooks.ts that calls
  rewriteCommandWithRtk() from the existing shared/rtk.ts module
- Handler is a no-op when RTK is disabled or not installed

* fix: correct import path for shared/rtk.js in register-hooks

* fix(tests): remove deleted pi-coding-agent/utils/rtk imports from execution seams test

The RTK rewrite logic was moved out of pi-coding-agent into the GSD
extension (bash_transform hook). Tests that directly imported the
deleted utils/rtk.ts are removed; remaining tests verify the shared
RTK module and GSD-layer surfaces that still call rewriteCommandWithRtk.
2026-03-26 09:33:07 -06:00

219 lines
9.8 KiB
JavaScript

#!/usr/bin/env node
// GSD Startup Loader
// Copyright (c) 2026 Jeremy McSpadden <jeremy@fluxlabs.net>
import { fileURLToPath } from 'url'
import { dirname, resolve, join, relative, delimiter } from 'path'
import { existsSync, readFileSync, mkdirSync, symlinkSync, cpSync } from 'fs'
// Fast-path: handle --version/-v and --help/-h before importing any heavy
// dependencies. This avoids loading the entire pi-coding-agent barrel import
// (~1s) just to print a version string.
const gsdRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..')
const args = process.argv.slice(2)
const firstArg = args[0]
// Read package.json once — reused for version, banner, and GSD_VERSION below
let gsdVersion = '0.0.0'
try {
const pkg = JSON.parse(readFileSync(join(gsdRoot, 'package.json'), 'utf-8'))
gsdVersion = pkg.version || '0.0.0'
} catch { /* ignore */ }
if (firstArg === '--version' || firstArg === '-v') {
process.stdout.write(gsdVersion + '\n')
process.exit(0)
}
if (firstArg === '--help' || firstArg === '-h') {
const { printHelp } = await import('./help-text.js')
printHelp(gsdVersion)
process.exit(0)
}
// ---------------------------------------------------------------------------
// Runtime dependency checks — fail fast with clear diagnostics before any
// heavy imports. Reads minimum Node version from the engines field in
// package.json (already parsed above) and verifies git is available.
// ---------------------------------------------------------------------------
{
const MIN_NODE_MAJOR = 22
const red = '\x1b[31m'
const bold = '\x1b[1m'
const dim = '\x1b[2m'
const reset = '\x1b[0m'
// -- Node version --
const nodeMajor = parseInt(process.versions.node.split('.')[0], 10)
if (nodeMajor < MIN_NODE_MAJOR) {
process.stderr.write(
`\n${red}${bold}Error:${reset} GSD requires Node.js >= ${MIN_NODE_MAJOR}.0.0\n` +
` You are running Node.js ${process.versions.node}\n\n` +
`${dim}Install a supported version:${reset}\n` +
` nvm install ${MIN_NODE_MAJOR} ${dim}# if using nvm${reset}\n` +
` fnm install ${MIN_NODE_MAJOR} ${dim}# if using fnm${reset}\n` +
` brew install node@${MIN_NODE_MAJOR} ${dim}# macOS Homebrew${reset}\n\n`
)
process.exit(1)
}
// -- git --
try {
const { execFileSync } = await import('child_process')
execFileSync('git', ['--version'], { stdio: 'ignore' })
} catch {
process.stderr.write(
`\n${red}${bold}Error:${reset} GSD requires git but it was not found on PATH.\n\n` +
`${dim}Install git:${reset}\n` +
` https://git-scm.com/downloads\n\n`
)
process.exit(1)
}
}
import { agentDir, appRoot } from './app-paths.js'
import { applyRtkProcessEnv } from './rtk.js'
import { serializeBundledExtensionPaths } from './bundled-extension-paths.js'
import { discoverExtensionEntryPaths } from './extension-discovery.js'
import { loadRegistry, readManifestFromEntryPath, isExtensionEnabled } from './extension-registry.js'
import { renderLogo } from './logo.js'
// pkg/ is a shim directory: contains gsd's piConfig (package.json) and pi's
// theme assets (dist/modes/interactive/theme/) without a src/ directory.
// This allows config.js to:
// 1. Read piConfig.name → "gsd" (branding)
// 2. Resolve themes via dist/ (no src/ present → uses dist path)
const pkgDir = resolve(dirname(fileURLToPath(import.meta.url)), '..', 'pkg')
// MUST be set before any dynamic import of pi SDK fires — this is what config.js
// reads to determine APP_NAME and CONFIG_DIR_NAME
process.env.PI_PACKAGE_DIR = pkgDir
process.env.PI_SKIP_VERSION_CHECK = '1' // GSD runs its own update check in cli.ts — suppress pi's
process.title = 'gsd'
// Print branded banner on first launch (before ~/.gsd/ exists).
// Set GSD_FIRST_RUN_BANNER so cli.ts skips the duplicate welcome screen.
if (!existsSync(appRoot)) {
const cyan = '\x1b[36m'
const green = '\x1b[32m'
const dim = '\x1b[2m'
const reset = '\x1b[0m'
const colorCyan = (s: string) => `${cyan}${s}${reset}`
process.stderr.write(
renderLogo(colorCyan) +
'\n' +
` Get Shit Done ${dim}v${gsdVersion}${reset}\n` +
` ${green}Welcome.${reset} Setting up your environment...\n\n`
)
process.env.GSD_FIRST_RUN_BANNER = '1'
}
// GSD_CODING_AGENT_DIR — tells pi's getAgentDir() to return ~/.gsd/agent/ instead of ~/.gsd/agent/
process.env.GSD_CODING_AGENT_DIR = agentDir
// RTK environment — make ~/.gsd/agent/bin visible to all child-process paths,
// not just the bash tool, and force-disable RTK telemetry for GSD-managed use.
applyRtkProcessEnv(process.env)
// NODE_PATH — make gsd's own node_modules available to extensions loaded via jiti.
// Without this, extensions (e.g. browser-tools) can't resolve dependencies like
// `playwright` because jiti resolves modules from pi-coding-agent's location, not gsd's.
// Prepending gsd's node_modules to NODE_PATH fixes this for all extensions.
const gsdNodeModules = join(gsdRoot, 'node_modules')
process.env.NODE_PATH = [gsdNodeModules, process.env.NODE_PATH]
.filter(Boolean)
.join(delimiter)
// Force Node to re-evaluate module search paths with the updated NODE_PATH.
// Must happen synchronously before cli.js imports → extension loading.
// eslint-disable-next-line @typescript-eslint/no-require-imports
const { Module } = await import('module');
(Module as any)._initPaths?.()
// GSD_VERSION — expose package version so extensions can display it
process.env.GSD_VERSION = gsdVersion
// GSD_BIN_PATH — absolute path to this loader (dist/loader.js), used by patched subagent
// to spawn gsd instead of pi when dispatching workflow tasks
process.env.GSD_BIN_PATH = process.argv[1]
// GSD_WORKFLOW_PATH — absolute path to bundled GSD-WORKFLOW.md, used by patched gsd extension
// when dispatching workflow prompts. Prefers dist/resources/ (stable, set at build time)
// over src/resources/ (live working tree) — see resource-loader.ts for rationale.
const distRes = join(gsdRoot, 'dist', 'resources')
const srcRes = join(gsdRoot, 'src', 'resources')
const resourcesDir = existsSync(distRes) ? distRes : srcRes
process.env.GSD_WORKFLOW_PATH = join(resourcesDir, 'GSD-WORKFLOW.md')
// GSD_BUNDLED_EXTENSION_PATHS — dynamically discovered bundled extension entry points.
// Uses the shared discoverExtensionEntryPaths() to scan the bundled resources
// directory, then remaps discovered paths to agentDir (~/.gsd/agent/extensions/)
// where initResources() will sync them.
const bundledExtDir = join(resourcesDir, 'extensions')
const agentExtDir = join(agentDir, 'extensions')
const registry = loadRegistry()
const discoveredExtensionPaths = discoverExtensionEntryPaths(bundledExtDir)
.map((entryPath) => join(agentExtDir, relative(bundledExtDir, entryPath)))
.filter((entryPath) => {
const manifest = readManifestFromEntryPath(entryPath)
if (!manifest) return true // no manifest = always load
return isExtensionEnabled(registry, manifest.id)
})
process.env.GSD_BUNDLED_EXTENSION_PATHS = serializeBundledExtensionPaths(discoveredExtensionPaths)
// Respect HTTP_PROXY / HTTPS_PROXY / NO_PROXY env vars for all outbound requests.
// pi-coding-agent's cli.ts sets this, but GSD bypasses that entry point — so we
// must set it here before any SDK clients are created.
// Lazy-load undici (~200ms) only when proxy env vars are actually set.
if (process.env.HTTP_PROXY || process.env.HTTPS_PROXY || process.env.http_proxy || process.env.https_proxy) {
const { EnvHttpProxyAgent, setGlobalDispatcher } = await import('undici')
setGlobalDispatcher(new EnvHttpProxyAgent())
}
// Ensure workspace packages are linked (or copied on Windows) before importing
// cli.js (which imports @gsd/*).
// npm postinstall handles this normally, but npx --ignore-scripts skips postinstall.
// On Windows without Developer Mode or admin rights, symlinkSync will throw even for
// 'junction' type — so we fall back to cpSync (a full directory copy) which works
// everywhere without elevated permissions.
const gsdScopeDir = join(gsdNodeModules, '@gsd')
const packagesDir = join(gsdRoot, 'packages')
const wsPackages = ['native', 'pi-agent-core', 'pi-ai', 'pi-coding-agent', 'pi-tui']
try {
if (!existsSync(gsdScopeDir)) mkdirSync(gsdScopeDir, { recursive: true })
for (const pkg of wsPackages) {
const target = join(gsdScopeDir, pkg)
const source = join(packagesDir, pkg)
if (!existsSync(source) || existsSync(target)) continue
try {
symlinkSync(source, target, 'junction')
} catch {
// Symlink failed (common on Windows without Developer Mode / admin).
// Fall back to a directory copy — slower on first run but universally works.
try { cpSync(source, target, { recursive: true }) } catch { /* non-fatal */ }
}
}
} catch { /* non-fatal */ }
// Validate critical workspace packages are resolvable. If still missing after the
// symlink+copy attempts, emit a clear diagnostic instead of a cryptic
// ERR_MODULE_NOT_FOUND from deep inside cli.js.
const criticalPackages = ['pi-coding-agent']
const missingPackages = criticalPackages.filter(pkg => !existsSync(join(gsdScopeDir, pkg)))
if (missingPackages.length > 0) {
const missing = missingPackages.map(p => `@gsd/${p}`).join(', ')
process.stderr.write(
`\nError: GSD installation is broken — missing packages: ${missing}\n\n` +
`This is usually caused by one of:\n` +
` • An outdated version installed from npm (run: npm install -g gsd-pi@latest)\n` +
` • The packages/ directory was excluded from the installed tarball\n` +
` • A filesystem error prevented linking or copying the workspace packages\n\n` +
`Fix it by reinstalling:\n\n` +
` npm install -g gsd-pi@latest\n\n` +
`If the issue persists, please open an issue at:\n` +
` https://github.com/gsd-build/gsd-2/issues\n`
)
process.exit(1)
}
// Dynamic import defers ESM evaluation — config.js will see PI_PACKAGE_DIR above
await import('./cli.js')