AI Agent Observability Under Clock Skew: Rebuild Event Order

An AI agent can produce a perfectly plausible trace whose timestamps tell an impossible story. A tool result appears before the request that caused it. A completion lands before the run starts. A delayed progress record arrives after the outcome and makes a finished run look active again. The practical answer is not to “fix” the timeline by sorting harder. Do not infer agent state from origin wall clock timestamps alone. Keep four pieces of evidence— event at , observed at , source seq , and depends on —and use each for the job it can actually support. Rebuild causal order from dependencies and per source sequence. Use the collector clock for evidence freshness. Measure durations with a monotonic clock inside one process. If a required predecessor is missing, the health verdict is uncertain , not stuck, healthy, or complete. That rule is small enough to test. The fixture below puts two kinds of clock disorder and one broken causal chain into eleven events. A deterministic audit recovers two workflows and refuses to invent an order for the third. A wall clock sort can reverse the work Distributed agent work crosses clocks: the runtime host, a tool server, a queue, a verifier, and the collector may all stamp the same run. Clock synchronization reduces their disagreement; it does not turn those clocks into one causal authority. NTP itself models clock offset, network delay, dispersion, and synchronization distance rather than promising identical time everywhere (RFC 5905). The first fixture workflow makes the problem visible. Its agent clock is 45 seconds fast, while its tool clock is 30 seconds slow. The actual dependency chain is: Sorting the same records by event at places g3 before g1 . A dashboard built on that order can calculate a negative tool duration, show a terminal result before the start, or mistake later arriving evidence for a new state transition. None of those conclusions follows from the work. They follow from comparing wall clocks that have different offsets. OpenTelemetry's stable Logs Data Model preserves the distinction needed here. Timestamp is when the event occurred according to the origin clock; ObservedTimestamp is when the collection system observed it (OpenTelemetry Logs Data Model). Keeping both is useful, but neither field is a universal ordering key: Field Safe use Unsafe inference event at Display source local time; correlate with host evidence Cross host causal order or latency observed at Collector relative freshness and ingestion delay Time the work actually happened source seq Order emitted by one source incarnation Order across unrelated sources depends on Explicit cross source causal edges Proof that an omitted event occurred monotonic elapsed Duration inside one process lifetime Comparable timestamp across machines The source incarnation matters. A counter must be scoped by something like (source id, boot id) , because a restarted process may begin again at sequence 1. A bare global looking integer invites a different false alarm: the collector reads an expected reset as replay or regression. The distinction between wall and monotonic time is also operational, not academic. Go's time package explains that wall clocks are subject to synchronization changes while monotonic clocks are for measuring time; values returned by time.Now can carry both readings so elapsed time operations remain robust when wall time changes (Go monotonic clocks). Other runtimes expose different APIs, but the decision stays the same: compute a local tool duration from a local monotonic interval, then export that duration as evidence. Do not subtract two unrelated machines' wall times and call the result latency. Rebuild causality before classifying health The event contract is deliberately compact: dependsOn creates the cross source edge from request to result. Consecutive sourceSeq values create local edges within one (source, bootId) stream. The audit combines those edges, checks for missing predecessors and sequence gaps, then performs a topological sort. Wall clock and collector time inversions become diagnostics attached to edges; they do not rewrite the graph. Run the artifact from its directory: Its fixed summary is: The first workflow is reconstructed despite the origin clock inversion. The second has a different failure of naive ordering: d3 reaches the collector before its predecessor d2 , so sorting by observed at reverses their dependency. The graph still recovers the intended order. The third workflow contains a tool result that names b missing request , an event absent from the evidence set. It also looks terminal before start under a wall clock sort, but the audit does not “repair” it by guessing. Its status is uncertain . That produces a useful decision order for agent health: 1. Validate identity. Reject duplicate event IDs and scope sequence numbers to a source incarnation. 2. Build local edges. Consecutive source sequence values establish emission order; a gap is evidence loss, not permission to close the gap. 3. Build cross source edges. Join requests, tool results, delegated work, approvals, and outcome checks with explicit predecessor IDs. 4. Reject invented certainty. A missing predecessor, a sequence gap, or a cycle makes the affected verdict uncertain. 5. Order the admissible graph. Topologically sort the complete portion; retain wall clock inversions as evidence about clock quality. 6. Classify state only now. Apply the working, waiting, stuck, and outcome rules to the causal order rather than arrival order. This keeps activity separate from useful progress. A late heartbeat may be fresh at the collector but causally older than an already verified outcome. It should not reopen the run. A tool result may be observed recently but depend on a request the collector never saw. It should not prove completion. A human approval event can legitimately leave a run waiting even if no new execution event follows; the dependency names the blocker. One implementation detail prevents many accidental regressions: make the health reducer monotonic where the workflow contract allows it. Once outcome invoice 42 is independently verified for run r7 , an older tool requested event cannot demote that outcome to “working.” It can update the evidence ledger, reveal delayed delivery, or raise a telemetry quality issue, but it cannot erase a stronger verified fact. Treat time as evidence with a boundary Causal reconstruction is not a clock synchronization replacement. You still need synchronized hosts for readable incident timelines, certificate validation, scheduler behavior, and operational correlation. The rule simply stops the health model from claiming more than those clocks prove. It also has four sharp limits. First, observed at is only authoritative relative to the collector that stamped it. Queueing, retries, backpressure, and collector failover can increase observed delay. Use it to ask “how long since this collector saw admissible evidence?” Do not automatically report observed at event at as network latency. Second, a dependency graph is only as complete as its instrumentation. A missing predecessor may mean packet loss, sampling, an exporter bug, or a producer that never emitted the event. The safe result is uncertainty plus a named evidence gap. It is not proof that the agent failed. Third, causal order does not verify the intended outcome. tool succeeded says the tool returned successfully under its own contract. It does not prove the file exists at the destination, the email reached the intended recipient, or the deployment serves the expected version. Keep outcome verification as a separate event with its own evidence. Fourth, topological order can be partial. Independent branches may have no meaningful order between them. Do not manufacture one for a prettier timeline. Present concurrent branches together, and require an explicit join or completion quorum before declaring the parent complete. The falsifiable claim for this article is narrow: for the supplied eleven event fixture, the audit must reconstruct two workflows, mark the broken chain uncertain, retain one origin time inversion and one collector time inversion as diagnostics, and expose both naive terminal before start cases. If any of those counts changes, the artifact fails. What this means for Sidewisp Sidewisp is intended to add a health layer around existing agent runtimes: distinguish working, waiting, stuck, uncertain, and outcome verified states using evidence with freshness and confidence. Clock aware causal evidence fits that direction because a green trace is not useful if its order was inferred from incompatible clocks. Sidewisp is currently in private preview. The public site and article system are live, but production agent health collection, runtime adapters, cron management, token cost analytics, and recovery are not generally shipped. This article describes an operating pattern and a test artifact, not a claim that Sidewisp currently reconstructs production timelines or fixes clock skew. The intended product is not a replacement runtime, mandatory gateway, raw tracing product, enterprise control plane, or autonomous fixer. It should sit beside existing agents, show uncertainty when the chain is incomplete, and keep human authority over any recovery action. If clock and delivery disorder are hiding the real state of your agents, joining the private preview is the restrained next step; it is not a promise that production monitoring is available today. The operational default is therefore simple: preserve origin time for context, collector time for freshness, monotonic elapsed time for local durations, and explicit edges for causality. When those edges are incomplete, say so. An honest uncertain is healthier than a beautifully sorted false story.