Coding Agent Observability Is Harder Than It Looks

Our approach to capturing agent traces

We collect traces on the device.

We chose device capture because we wanted to connect sessions to code changes. Git lives where the agent runs. Through a gateway, you might see the model propose an edit to src/app.ts, but you can't tell which repository contains that path, which branch the engineer has checked out, or whether the agent applied the edit.

Agents also change code through shell commands, MCP tools, custom commands, and scripts that run several steps. To reconstruct those changes from requests, you'd need to parse tool-specific patch formats and shell output, then check whether your reconstructed diff matches the files on disk.

A gateway adds an operational concern: engineers depend on it to reach their models. We wanted collection failures to leave engineers' agent sessions unaffected.

Device capture applies to cloud sandboxes and dev containers too. You have a file system and a Git checkout in those environments, with local changes you won't see through the model provider.

Hooks first, transcripts alongside them

On the device, we use hooks as our primary source and fill in gaps from session transcripts.

Vendors expose hooks so third parties can extend their harnesses. That gives vendors a reason to keep them stable, though it falls short of a formal data contract. The names differ, but the events overlap: session starts and ends, user prompts, tool calls, and sub-agent launches.

Through a hook, we get a structured event that identifies a user prompt or a failed tool call. With a proxy or raw log, we'd have to infer that from the content.

The timing matters, too. At the moment of an edit, we record the repository, branch, and commit through auspex. If we waited for the harness to flush a transcript and for the collector to read it, the engineer might have switched branches.

We need transcripts for information such as token usage. The split varies by tool: some vendors include rich detail in hooks and little in session logs; others do the reverse. We choose the best source for each tool and join the records.

Keeping up with undocumented formats

Coding agent vendors don't publish data contracts for their hook payloads or session logs. A vendor can rename a field in a patch release, move data from a hook to a transcript, or add an event type without treating it as a breaking change.

You also have to account for variants within a product. Cursor's IDE and headless CLI produce different data, as do Copilot's CLI and Copilot Chat in VS Code. Some cloud agents skip hooks on a cold start.

auspex supports Claude Code, Cursor, Codex CLI, GitHub Copilot CLI, VS Code Copilot Chat and OpenCode today, with Pi coming soon. We maintain a separate integration for each and track their releases. Adding a tool means taking on that maintenance work for as long as engineers use it.

Keeping collection out of the engineer's way

To capture Git state at the moment of an edit, we have to run a hook before the agent continues. Engineers feel any delay on each turn.

Some attribution tools compare repository state before and after each tool call inside the hook. You get detailed results, but the work grows with repository size. In a large monorepo, that can mean waiting on collection during each turn.

We keep the hook's work small. Through auspex, we attach the repository, branch, and commit to the event, write it to a local folder, and exit under strict timeouts. We make no network calls from the hook. In a per-user background process, we redact, buffer, and upload the events. If the machine goes offline, we keep them on disk until it reconnects.

We also avoid kernel and system extensions, and we write nothing into your repositories, including Git notes. Engineers shouldn't have to change how they work to accommodate collection.

Attributing code across concurrent sessions

Matching an agent's edits to a pull request gets complicated when an engineer runs ten sessions against one repository. Those sessions may span worktrees and branch switches. Some use edit tools; others change files through shell scripts.

We do the matching in Span's backend. On the device, we record each session's actions and location. In the pipeline, we compare those edits with the pull request's diff and use the order of file changes to work out attribution. We match against the PR itself, so we don't need metadata to survive a squash or rebase. You don't need to install anything extra in your version control provider either.

Working in the backend also gives us more information to resolve overlap. If several sessions touched a file, we send the candidates upstream and compare them with the pull request, review, and ticket. We don't have to pick a session on the device with less context.

We give up some precision with this approach. If an engineer and an agent edit the same branch at the same moment, we can't separate their contributions as well as a tool that diffs on each keystroke. We expect this overlap to become less common as engineers alternate between longer stretches of agent work and human work. That's an assumption behind our design, and a tradeoff to consider for your team.

We're also rolling out Snapshotting to capture edits outside edit tools. During a session, we take lightweight snapshots of the Git working tree through auspex, in the background and in its own scratch space. With those snapshots, we can credit changes made through shell commands, scripts, MCP tools, and skills to the agent.

Making the data comparable

To compare skill usage or find turns where agents get stuck across an organization, you need a common schema. Each tool has its own event format.

We normalize the data in two stages. On the device, we map hook payloads to common attributes through auspex. In the pipeline, we account for gaps in each tool's data and reconstruct sessions in a shared format. We update both stages as vendors change their formats, so you can query sessions across tools without learning each vendor's JSON structure.