AI Agent Observability: Sample Traces, Keep Health Evidence

AI agent observability should not require retaining every trace, but it should never let trace sampling decide whether a critical health event exists. Sample the heavy diagnostic path when volume or cost demands it. Send compact, typed health receipts—outcome failure, approval wait, tool failure, runtime unreachable, and missing deliverable—through a separate path with its own delivery and freshness checks. That split solves a specific problem. A trace is excellent evidence for why a run behaved as it did. It is not the only place from which an operator can learn that the expected work is missing. A runtime that cannot be reached may emit no trace at all. A session waiting correctly for approval may have no error span. A run can finish with OK spans while an external verifier finds that the requested file does not exist. The reasonable default is therefore two retention policies, not one clever sampler: apply head or tail sampling to diagnostic traces, and retain every mandatory health receipt for a shorter, explicitly bounded period. Join the two records with a stable run ID when a trace is available. Do not store raw prompts or unrestricted tool payloads merely to keep the health verdict. Sample traces; keep health receipts OpenTelemetry's sampling documentation uses a narrow and useful definition: a sampled trace is processed and exported; a trace that is not sampled is not processed or exported. Head sampling decides early, commonly from a trace ID and desired percentage, without inspecting the whole trace. Tail sampling waits for more or all of the trace and can keep traces by error, latency, attributes, or other criteria. Those mechanisms optimize a population of traces. An agent health policy answers a different question: which observations are required to decide whether work is healthy, waiting, stuck, unreachable, or falsely complete? Record Purpose Sampling default Example evidence Diagnostic trace Explain execution and debug a selected run Sample when justified by volume and cost spans, tool duration, model calls, error path Health receipt Preserve a state changing operational fact Keep every mandatory kind outcome predicate failed, approval required, heartbeat missed Aggregate metric Measure rates and capacity Aggregate before storage where possible retry rate, queue depth, receipt loss Sensitive payload Reproduce content only under a separate authority Do not collect by default prompt, tool arguments, raw response The distinction does not make tail sampling unhelpful. A tail rule that always keeps traces containing an ERROR span is valuable. It still cannot keep a trace that never arrived, and it cannot infer that a successful looking trace failed an external deliverable check. The Collector's tail sampling processor documentation is explicit about an important prerequisite: all spans for a trace must reach the same collector instance for an effective decision. Treat that prerequisite as a boundary, not a flaw. Trace sampling operates after trace evidence exists. Health receipts must also cover failures outside that path. Define the unsampled health contract Keep the receipt small enough that retaining every mandatory kind is ordinary, not heroic. A useful record needs identity, semantics, evidence freshness, and source—not a transcript: The two clocks matter. The stable OpenTelemetry Logs Data Model defines Timestamp as the time an event occurred at the source and ObservedTimestamp as the time the collection system observed it. Preserve both semantics in a health receipt. A delayed receipt may still describe a real failure, but its delivery delay and evidence age should remain visible. Use a short mandatory kind registry owned by the workflow, not by the storage vendor. A starting set for agent operations might be: outcome failed : a task specific verifier rejected the promised result; approval wait : the run has a typed human dependency and an owner; tool error : a required tool operation failed after its bounded retry policy; runtime unreachable : an external observer could not reach the runtime by the expected deadline; missing deliverable : the run declared completion but the expected artifact was absent. The producer is part of the contract. An outcome verifier may emit outcome failed ; a runtime adapter may emit approval wait ; an external scheduler or heartbeat observer must emit runtime unreachable . Requiring an unreachable runtime to report its own unreachability is a circular design. For each receipt, apply four checks before it changes health: 1. Deduplicate by receipt id or a stable event key. 2. Validate schema version , kind , run id , and producer authority. 3. Compare occurred at and observed at against per kind freshness limits. 4. Update health by explicit precedence, preserving unknown when required evidence is missing. Do not let a receipt authorize an irreversible action. It can open an issue, route a wait, or prepare a bounded response. Recovery still needs the relevant approval boundary and a new observation that proves useful progress or the expected outcome. Replay the retention decision The retained fixture contains forty synthetic runs and five deliberately different critical cases. Deterministic ten percent head sampling hashes each trace ID. The tail policy keeps traces whose span status is ERROR . The third policy retains every registered health receipt. Run it with: The fixed result is: The head sample keeps runs 1, 3, 10, 25, 39 . Run 25 is the tool error case, so four other critical cases are absent from its sample. The ERROR only tail rule also keeps the tool error. It misses an outcome failure carried by an OK trace, an approval wait carried by an UNSET trace, a missing deliverable carried by an OK trace, and the unreachable runtime that produced no trace. This is a policy test, not a statistical result. The five critical runs were deliberately distributed so the replay contains both a retained case and missed cases. It does not claim that ten percent sampling usually captures one fifth of incidents, or that these five event kinds have equal frequency. Change the trace IDs, sampling rule, or fixture and the counts can change. What should not change is the acceptance criterion: every mandatory receipt kind must survive the health path, including a no trace case. Add a new operational verdict only after adding its producer, schema, fixture, retention rule, and loss monitor. Monitor the receipt path and bound retention An unsampled channel can still fail. Queue overflow, schema rejection, expired credentials, clock errors, producer bugs, and storage outages can make health evidence disappear. Monitor the channel with signals that do not depend solely on the channel itself: expected receipt counts by producer and workflow slot; producer heartbeat and last successful delivery time; rejected, duplicated, and delayed receipt counters; collector queue capacity and drop counters; periodic end to end canaries with a known receipt ID; reconciliation between scheduled runs, declared completions, and received outcomes. Absence must not become green. If an outcome verifier has not reported for a run that requires it, mark the outcome signal unavailable or uncertain. If the external observer itself is stale, do not claim the runtime is reachable. A health layer must expose gaps in its own evidence. Retaining every health receipt also does not mean retaining it forever. Choose retention from the operating decision: long enough to investigate, reconcile late evidence, and audit an approved intervention. Aggregate older counts when individual records are no longer needed. Remove or hash local paths, user content, prompt text, tool payloads, and credential material before export. Store a digest or bounded error class when it supports the decision. The trade off is explicit. A compact mandatory channel costs extra engineering and duplicates a small amount of trace metadata. In return, trace volume controls cannot silently erase the facts that drive health. Tail sampling remains useful for selecting diagnostic detail around known errors; a mandatory receipt gives you the incident even when the trace status is successful, incomplete, or absent. Before adopting the pattern, replay real sanitized cases from one workflow: healthy completion, false success, approval wait, tool failure, missed heartbeat, and collector outage. Verify both sides. The trace policy should meet its cost target, and the receipt policy should retain every required verdict without collecting sensitive payloads. Sidewisp's product direction is a health layer around existing agent runtimes: evidence, freshness, useful progress, waiting states, verified outcomes, and explicit approval boundaries. It is not a replacement runtime, mandatory gateway, raw tracing product, or autonomous fixer. Sidewisp is currently in private preview. The public site and article system are live, while production agent health collection, runtime adapters, cron management, token cost analytics, and recovery are not generally shipped. If this evidence retention boundary matches the failure modes you need to operate, you can join the private preview and describe the runtime and receipt kinds that matter. Primary references OpenTelemetry: Sampling — trace sampling terminology, head and tail sampling, and operational trade offs; reviewed July 26, 2026. OpenTelemetry Collector Contrib: Tail Sampling Processor — trace grouping, policy types, collector affinity, dropped traces, and late spans; reviewed July 26, 2026. OpenTelemetry: Logs Data Model — stable Timestamp and ObservedTimestamp semantics; reviewed July 26, 2026.