ADR-009: Trace artifact format and dispatch-observer seam

Status: Accepted (capture + token report + replay + token budget + offline viewer all shipped)

Context

The server can drive an app but cannot record what it did. An agent (or a human debugging one) has no portable record of which tools ran, in what order, with what inputs and outputs, or where the token budget went. The trace capability is the first product differentiator and the first genuine consumer of the plugin model (ADR-004): it must observe the WHOLE session, not just add isolated tools. Two new decisions are needed — how a plugin observes every tool call, and what the recorded artifact looks like.

Decision

1. A dispatch-observer seam on the dispatcher

The dispatcher (ADR-008) gains a set of best-effort observers. After every dispatch() resolves — success, validation failure, or unknown tool — it notifies each observer exactly once, through a single internal funnel, with a DispatchRecord:

{ tool, args, result, startedAt, finishedAt }

args is the parsed input (or the raw input when validation failed); result is the exact agent-facing envelope (so estimated_tokens is read from result._meta). Observers register via Dispatcher.addObserver(observer): () => void (returns an unsubscribe), also surfaced to tool handlers as ToolContext.addDispatchObserver so a plugin tool (e.g. trace_start) can attach a sink without the core depending on the plugin.

Observers are best-effort and must not throw: a throw is caught and logged, never propagated to the agent (same contract as the transport's console/dialog listeners). With zero observers the funnel is a single size > 0 check — no per-call overhead.

2. The trace artifact is JSONL, schema version 1

One JSON record per line. The first line is a meta header; each subsequent line is a call:

{ "v": 1, "kind": "meta", "started_at", "core_version" }
{ "kind": "call", "tool", "ok", "code"?, "started_at", "finished_at", "elapsed_ms",
  "estimated_tokens", "args", "result" }

JSONL was chosen over zstd/protobuf/OpenTelemetry: it is append-friendly, human-readable, and parses with JSON.parse per line — no dependency, no schema compiler. The explicit v lets replay, budget metadata, and viewer features evolve the shape compatibly.

3. Trace is a plugin, recording is opt-in

Per the lean-core thesis (ADR-007), trace ships as @electron-stagewright/plugin-trace, not core. It records only between trace_start and trace_stop, to an operator-chosen path, and skips its own trace_* calls. Records are buffered in memory (bounded by maxRecords) and written on stop, so the observer stays a cheap array push on the dispatch hot path; a crash before stop loses the buffer (streaming is a forthcoming improvement).

Rationale

• A core observer seam is the only way a plugin can see other tools' calls; making it a plugin keeps the lean-core tool surface intact while proving plugins can be session-wide.
• Recording all outcomes (including errors) makes a trace a faithful session record.
• Token reporting reads _meta.estimated_tokens, already computed per envelope (ADR-006), so the report needs no new measurement.

Alternatives considered

• Wrap each tool handler instead of an observer — plugins only add tools; they cannot wrap core tools, and a wrapper would not see validation failures (which fail before the handler).
• Stream each record to disk — more crash-durable but blocks the dispatch path or needs an ordered async queue; deferred behind the in-memory buffer + cap.
• Binary/OTel artifact — heavier dependency and tooling than the debugging use case needs.

Consequences

• ToolContext gains addDispatchObserver; DispatchRecord / DispatchObserver are public.
• A trace captures inputs/outputs that may contain sensitive data (typed text, eval payloads). It is opt-in and operator-pathed (same trust model as screenshots/logs); a redact config drops named argument fields. Broader redaction/sandboxing is a forthcoming follow-up.
• The initial plugin surface ships trace_start, trace_stop, trace_tokens, trace_status. Replay, budget reporting, and the visual viewer extend that surface compatibly.

Related decisions

• ADR-004 (plugin model) — the contract the trace plugin is built on.
• ADR-006 (error code registry) — estimated_tokens and the namespaced trace.* error codes.
• ADR-008 (server and tool dispatcher) — the dispatch path the observer seam funnels through.

References

• packages/core/src/server/dispatcher.ts — observer set, addObserver, the #complete funnel.
• packages/core/src/tools/types.ts — DispatchRecord, DispatchObserver, the ctx seam.
• packages/plugin-trace/ — the plugin, recorder (JSONL + token summary), and tests.

Status Update (2026-06-04) — the seam becomes bidirectional; trace_replay

The original seam made plugins OBSERVE every dispatch. Replay needs the inverse — to RE-DISPATCH recorded calls — so the seam gains an active half:

• ToolContext.dispatch(tool, args): Promise<ToolResult> re-dispatches a tool through the same dispatcher (full Zod validation, operation-type routing, session context, observer notification), and ToolContext.validate(tool, args): ErrorResponse | null checks a call against the current schema WITHOUT running the handler. Both are wired by the dispatcher; the core still does not depend on the plugin.
• Bounded recursion. Re-dispatch depth is tracked in an AsyncLocalStorage<number> (per async chain, so concurrent dispatches never share a counter); past MAX_REDISPATCH_DEPTH = 1 a re-dispatch returns BAD_ARGUMENT instead of recursing. A tool may drive other tools (replay at depth 0, its calls at depth 1); a re-dispatched tool may not re-dispatch again.
• trace_replay reads an artifact and re-dispatches each call in order. The crux is session-id remapping: the recorded run's session ids are defunct, so the engine keeps a recorded → new map (learned from each result's _meta.session_id) and rewrites each call's sessionId arg before dispatch. Divergence is judged on the STABLE outcome (ok + error code), not the full envelope (payloads vary run-to-run); each diverged call carries a bounded, size-capped field-level diff. dryRun re-validates against current schemas without dispatching (detects a tool whose signature drifted since recording). A trace recorded with redact cannot be faithfully replayed — redacted args diverge — which the tool reports rather than hides.
• The "replay a deterministic minimal-app session" acceptance criterion lands here, proven by a gated real-Electron smoke (record launch→snapshot→click→stop, then replay with all matched).
• Still forthcoming at this point in the decision history: the visual offline viewer and token-budget enforcement.

New references: packages/plugin-trace/src/replay.ts (engine); ToolContext.dispatch / validate and the REDISPATCH_DEPTH guard in dispatcher.ts.

Status Update (2026-06-04) — token budget + a pre-dispatch guard seam

The trace plugin gains a token budget, and the dispatch seam gains a third capability — a pre-dispatch VETO — alongside observe and re-dispatch:

• ToolContext.addDispatchGuard(guard) registers a DispatchGuard that runs for every dispatch BEFORE the handler (after validation + operation-type routing). Returning a ToolResult vetoes the call (handler never runs; the envelope is returned and observers still see it via #complete); returning null allows it. Guards run in registration order, first veto wins, and a throwing guard is caught, logged, and treated as allow (fail-open — a guard bug must never wedge the tool surface). This is the active-enforcement counterpart to the observe-only seam.
• Budget. trace_start accepts budgetTokens (+ warnThreshold, default 0.8); the recorder tracks exact spent — counted BEFORE the record-cap drop, so an overflowed trace still reports the true total — and exposes budgetStatus() ({ budget_tokens, spent, remaining, over_budget, near_budget, warn_threshold }). The budget (and exact spent) persist in the JSONL meta header (additive optional fields; format version stays 1). trace_status, trace_tokens, the new trace_budget, and trace_stop all surface the nested budget object.
• Enforcement is opt-in. By default the budget is advisory (report only; the agent polls trace_budget and self-limits). trace_start({ enforce: true }) (or the enforceBudget config) registers a guard that vetoes over-budget calls with the new trace.BUDGET_EXCEEDED (http 429, not retryable, carries next_actions). The guard skips trace_* so an over-budget agent can always trace_stop to recover; the call that tips the budget over still runs (its cost is unknown beforehand), everything after it is blocked. The guard is released on stop + teardown.
• Still forthcoming: the visual offline viewer.

New references: DispatchGuard / DispatchGuardCall / ToolContext.addDispatchGuard and the #guards set in dispatcher.ts; BudgetStatus / budgetStatusOf and budget tracking in packages/plugin-trace/src/recorder.ts.

Status Update (2026-06-06) — the offline viewer

The last forthcoming piece — a visual viewer — ships as trace_view. The format decision the original scope left open ("viewer format chosen before implementation") is resolved as a single self-contained HTML document: inline CSS and JS, no external assets, no CDN, no runtime server, no build step. The operator opens it by double-clicking; it works fully offline and is trivial to attach to a bug report. This was chosen over a hosted/served viewer (needs a process and a port), a framework SPA (build step + bundle), or a terminal viewer (not shareable) because a trace is a portable record — the viewer should be as portable as the artifact.

• trace_view({ path, out? }) reads a written artifact (the shared loadTrace helper, so a missing file → trace.ARTIFACT_NOT_FOUND, malformed JSONL → trace.ARTIFACT_INVALID), renders it, and writes the HTML (a write failure → the existing trace.ARTIFACT_WRITE_FAILED). With no out, the report is written next to the trace with a .html extension. The render is a pure function (renderTraceHtml(parsed, { generatedAt? })), so it is unit-tested without disk I/O, and the generation timestamp is injected for deterministic output.
• The report shows summary cards (calls, ok/error, total estimated tokens), a token-budget bar when the trace carries a budget, the largest-response and per-tool token tables (reusing summarizeTrace), and an expandable timeline of every call (native <details>, so it is readable with JS disabled). A small fixed inline script adds tool-name filtering and expand/collapse-all as progressive enhancement.
• SECURITY. A trace captures arbitrary tool inputs/outputs (typed text, eval payloads, app content), so every dynamic value is HTML-escaped on render (text AND attribute values). The inline script is a fixed constant with no trace data interpolated into it, so it adds no injection surface. A unit test asserts a captured <img onerror=…> / </script> payload is escaped rather than emitted as live markup.

New references: packages/plugin-trace/src/viewer.ts (renderTraceHtml / escapeHtml); the trace_view tool in packages/plugin-trace/src/index.ts.

Status Update (2026-07-13) — streaming JSONL v2 and crash recovery

The original v1 buffer-on-stop tradeoff is replaced for new recordings. Version 2 keeps JSONL's human-readable, dependency-free format while making trace capture recoverable after a process or host failure:

• Lifecycle. trace_start first creates <path>.partial and writes a header record (format: "stagewright-trace", version: 2). Each dispatch appends a sequenced call record. A clean trace_stop drains the writer, appends a footer with complete:true, the persisted call count, overflow state, and exact budget spend, then atomically renames the partial file to the requested path. An optional fsync plugin setting syncs before that rename when an operator prefers durability over stop-time throughput.
• Recovery and compatibility. A v2 reader accepts a .partial file without a footer and returns complete:false; it also ignores only a final, unterminated JSON fragment, the expected shape if execution ended during an append. It rejects malformed ordering, duplicate or skipped sequence numbers, and lying footers. The v1 meta/call reader remains separate so a v2 file is never presented as v1-compatible.
• Bounded observer path. Dispatches redact and enqueue one record without awaiting disk I/O. The queue is bounded by the configured persisted-record cap, preserves write order, and retains only aggregate token data plus a bounded largest-response set for live reporting. A write error leaves the partial artifact available, surfaces trace.ARTIFACT_WRITE_FAILED, and never reports a false successful final artifact.
• Consumers. Replay, token reporting, and the offline viewer consume the shared parsed v1/v2 projection. They can inspect partial evidence; only a footer-complete v2 artifact is published at the requested final path.

New references: packages/plugin-trace/src/recorder.ts (streaming writer, v1/v2 readers); packages/plugin-trace/src/index.ts (start/stop lifecycle); packages/plugin-trace/src/viewer.ts (partial-artifact disclosure).

Status Update (2026-07-13) — promoted replay specifications

Raw replay intentionally remains a diagnostic comparison of stable ok/error-code outcomes. A new trace_promote path turns selected trace calls into a compact stagewright-replay v1 JSON specification for review and eventual automation:

• Promotion retains only call args and explicit ok checkpoints; it does not copy arbitrary response payloads into a committed contract. Recorded session ids become deterministic placeholders that a runner resolves from the fresh launch result.
• Promotion redacts password, token, authorization, and cookie argument fields by default and accepts additional scoped args.* redactions before it writes the spec. The same configured result redactions are applied to mismatch reports.
• The pure spec engine supports intentional result checks (exact, subset, regex, ignore) and numeric tolerance. Session ids, timestamps, and absolute paths can be normalized so they do not create false positives. Regex checks reject oversize or nested-quantifier sources before execution and cap their input.

New references: packages/plugin-trace/src/spec.ts (schema, promotion, oracle engine) and the trace_promote tool in packages/plugin-trace/src/index.ts.

Status Update (2026-07-13) — headless replay runner

@electron-stagewright/plugin-trace now publishes an electron-stagewright-replay executable for CI. It reads one promoted spec, starts a fresh core server with the requested plugin/eval/profile configuration, runs the existing dispatcher path, and emits either a compact human report or one machine-readable JSON object. Exit codes distinguish checkpoint mismatch, malformed spec, failed app launch, infrastructure, and CLI usage. Filters preserve the session creator of a selected dependent step, so narrowing a run never leaves a stale placeholder. The package smoke packs the plugin, installs it outside the monorepo beside core, and proves the published bin against a real Electron app.