AI Agent Observability: Fingerprint Every Run

The shortest useful answer is: attach a run manifest fingerprint to every agent run, then compare health only after you know whether the runtime, model snapshot, prompt, policy, tool schema, and deployment image were the same. A trace tells you what a run did. The manifest tells you which version of the system did it. That fingerprint is not a health score and it does not prove why a run failed. It is a branching key. If a regression appears under a new fingerprint, inspect configuration drift first. If it appears under the old fingerprint, look for unrecorded environment, dependency, provider, data, or permission changes. A stable agent name can hide a different system Suppose support triage completes two tickets correctly on Friday and misses a required escalation on Monday. Both runs have the same agent name. Both emit heartbeats. Both call tools. Grouping them together feels natural—and can be wrong. Between those runs, any of these may have changed: Manifest component Record Avoid recording Runtime name and exact version or commit host path, access token Model provider and pinned snapshot when available API key, full response Prompt SHA 256 of the reviewed prompt bundle raw customer or system prompt Policy hash or immutable revision secret values embedded in policy Tools canonical schema hash credentials, tool payloads Deployment image digest or source commit registry password This follows an established observability idea rather than inventing a second trace format. The OpenTelemetry Resource SDK describes a resource as an immutable representation of the entity producing telemetry. For an agent run, the execution configuration is part of that identity. Keep the trace ID for one execution and keep the manifest fingerprint for the version that produced it; neither replaces the other. A useful manifest is deliberately boring. It contains stable identifiers, not observations such as latency, token count, exit status, or outcome. Those belong beside the manifest. Mixing them into the hash would create a new fingerprint for every run and destroy the comparison. There is also a privacy boundary: hash a prompt or policy locally, but do not upload the original merely to explain its identity. Hashes can still leak information when the input comes from a tiny guessable set, so use opaque release IDs or a keyed local digest when that threat matters. Never put credentials in the manifest, even before hashing the whole object. Canonicalize before hashing Hashing raw JSON is a trap because object key order and insignificant serialization choices can differ while the represented configuration stays the same. The JSON Canonicalization Scheme in RFC 8785 defines deterministic serialization, including recursive property sorting. Production code should use a reviewed JCS implementation when interoperability matters. For a compact local experiment, the following Node.js function is enough for ordinary finite JSON values: The experiment used Node.js v22.23.1 . It is intentionally narrower than RFC 8785: it sorts object keys recursively and rejects non finite numbers, but it is not a claim of complete cross language JCS compliance. That limitation belongs next to the code, not in a footnote after a reader has copied it. The manifest itself can remain small: The placeholder values above are fixture identifiers, not vendor claims. In a real collector, derive them on the host from pinned releases. The broader principle resembles SLSA provenance: retain verifiable information about where, when, and how something was produced. A run manifest borrows that principle for diagnosis; it is not automatically a SLSA attestation. A six run replay exposes both the value and the limit I tested the rule against six synthetic NDJSON records for one agent. Runs 101 and 102 contain the same manifest values in different JSON key order. Run 103 changes only the prompt hash, 104 changes only the model snapshot, and 105 changes only the tool schema hash. Run 106 fails because of a dependency outage that the fixture manifest does not represent. The audit command was: The deterministic result: Two observations matter more than the counts. First, one agent name concealed four recorded configurations. Second, the two differently ordered baseline objects produced the same SHA 256 fingerprint, so serialization order did not create false drift. The counterexample is the important part: run 106 failed with the baseline fingerprint. An unchanged fingerprint did not make the run healthy, and it did not prove the environment was unchanged. It only showed that the recorded configuration fields were unchanged. The next investigation should examine evidence outside that manifest—provider availability, input data, network route, permission state, and dependency freshness. The fixture is synthetic, so it demonstrates the mechanics and falsifies an overclaim; it does not estimate how often configuration drift causes real incidents. That would require production data with controlled releases and verified outcomes. Turn the fingerprint into a triage decision Use the fingerprint only after the run has an observable outcome. Activity alone—tokens emitted, tools called, or a process still alive—does not establish useful progress. Outcome evidence Fingerprint comparison First branch Passed Same as baseline Keep as a comparable healthy reference Failed Changed Diff the redacted manifest fields; consider a bounded rollback or replay Failed Same Inspect dependencies, permissions, inputs, provider state, and missing manifest coverage Unknown Either Collect or define the expected deliverable before diagnosing drift Three operating rules keep this honest: 1. Freeze the comparison point. Choose a verified successful run for the same task class, not merely the most recent green status. 2. Keep a redacted field level map. The whole manifest hash says “different.” Individual component digests say where to inspect without revealing content. 3. Verify the outcome after intervention. A rollback command succeeding is activity. Recovery means the expected deliverable appears, the test passes, or another deterministic acceptance check clears. Do not automatically roll back every changed fingerprint. A release may be intentional, and a failure may come from the input rather than the release. Use the change as evidence for a review, preserve human approval for consequential actions, and record what happened after the action. Where this fits in agent health monitoring The run manifest fingerprint connects three health questions that traces alone cannot resolve cleanly: did a decision policy change, did a tool contract change, and did outcome quality regress under the same recorded system? It also improves incident handoffs because another operator can compare exact release identity without receiving prompts, secrets, or raw tool payloads. Sidewisp's intended health model includes execution, memory and context, tools, outcomes, availability, and cost. A future host side adapter could use a redacted manifest as supporting evidence for those diagnoses, but that is planned territory—not a shipped monitoring claim. Sidewisp is currently in private preview. The public site and article system are live; production agent health collection, runtime adapters, and automated recovery are not generally available. If this evidence model matches a failure you operate today, the restrained next step is to join the private preview waitlist and describe the runtime and outcome check you need—not to assume Sidewisp already collects it.