AI Agent Observability for Retries: Catch Duplicate Effects

An agent should not retry a tool call merely because its trace ends in a timeout. The request may have reached the provider, changed real state, and lost only the response. A second attempt can then send the message twice, create two tickets, or provision two resources while both traces look individually reasonable. The useful health rule is stricter: one logical operation must produce no more than one verified effect . Give the operation a stable identity, keep that identity across attempts, record provider side effect receipts, and block automatic retry when the effect cannot be looked up safely. Attempt counts and HTTP status still help with diagnosis, but neither proves the outcome. This article builds that rule into a small effect ledger and tests it against four operations. The fixture contains eight attempts and four provider observations. An attempt only policy would see timeouts and keep retrying three operations. The effect aware audit instead finds one healthy replay, one duplicate effect incident, one idempotency key conflict, and one honestly uncertain operation. A timeout is not proof that nothing happened The dangerous window sits between remote execution and local acknowledgement. A provider can commit an effect and then lose the response on the way back. From the agent's side, these two histories are observationally similar: 1. the request never reached the provider; 2. the request completed, but the response did not reach the agent. Only the first history is safe to repeat without another safeguard. The second creates a duplicate when the operation is not naturally idempotent. HTTP semantics provide a useful boundary. RFC 9110 defines an idempotent method as one whose intended server effect is the same after multiple identical requests as after one request. It permits automatic repetition after a communication failure for idempotent methods, but says a client should not automatically retry a non idempotent request unless it knows the operation is effectively idempotent or can detect that the original was never applied. That distinction belongs in agent health. A PUT that replaces a known record and a POST that sends an email can both time out, yet they do not have the same retry boundary. A generic “timeout → retry” policy erases the semantic fact that matters most. Provider support helps, but the contract must be read precisely. Stripe documents that it stores the status code and body for the first request made with an idempotency key, then returns that result to later requests with the same key. It also compares parameters and rejects reuse with different parameters. The key is therefore not a random label attached to each attempt. It represents one stable logical operation. Amazon EC2 documents a similar client token pattern: a successful retry with the same token and parameters performs no further action, while changed parameters can produce IdempotentParameterMismatch . EC2 also scopes some guarantees regionally or zonally. “Has a token” is not enough; the health record needs the token's scope, payload identity, retention window, and provider behavior. The reasonable default is: reuse one operation identity across all attempts; reuse the provider's idempotency key only for the same canonical payload; after an ambiguous result, query by that identity before retrying; if the provider offers neither idempotency nor lookup, require a human decision for consequential effects. Backoff reduces pressure on a failing service. It does not turn a non idempotent action into an idempotent one. Record effects, not only attempts An ordinary trace answers “what did the agent try?” An effect ledger answers the different question “what durable change can we prove?” Keep the two records linked, because attempts remain useful evidence, but do not treat an attempt's terminal span as the business outcome. A minimal ledger needs these fields: Field Purpose Health failure it exposes operationId Stable identity for the user's intended operation A new ID generated for every retry attemptId Identity for one transport attempt Missing or overlapping attempts idempotencyKey Provider deduplication identity, when supported Key changes across retries payloadHash Hash of a canonical, redacted payload Same key reused for different intent effectRef Provider or destination identity of the actual effect More than one durable effect result Transport observation such as timeout or success Ambiguous acknowledgement observedAt Time the evidence was collected Stale evidence mistaken for current state Do not put secrets, email addresses, full prompts, or raw tool payloads in these fields. Hash a canonical representation after removing volatile values. Keep the raw sensitive material at the origin when investigation requires it. The operation identity should be minted when intent becomes durable, not inside the retry loop. For example: The snippet is incomplete by design: catching an exception and continuing is not proof of safety. The caller must also retain the provider's returned object ID or query the provider by the same business identity after an ambiguous response. Count unique effects, not successful responses. Two successful responses that both name ticket 908 describe one effect. One timeout followed by a success that names delivery a and delivery b describes two effects. Conversely, zero receipts does not prove zero effects when the lookup channel is unavailable. That state is uncertain , not healthy and not automatically stuck. Payload identity is a separate gate. If two attempts share an idempotency key but have different canonical payload hashes, stop before interpreting effect count. The caller may have accidentally reused a key after changing the requested region, recipient, amount, or resource shape. Provider side parameter mismatch errors are useful evidence of this exact fault. Run a four operation effect audit The inspectable fixture used for this article is NDJSON. Each line is either an attempt or an effect observation. The complete local artifact contains four logical operations: op ticket 42 : two attempts share one key and one payload; both observations point to ticket 908 ; op webhook 77 : two attempts have no idempotency key and reveal delivery a plus delivery b ; op vm 5 : two attempts reuse one key with different payload hashes; op email 3 : two attempts time out, no effect receipt is available, and the provider has no lookup path. The audit groups records by operationId , rejects payload drift before counting effects, and counts distinct effectRef values rather than effect observation rows: Running the repository artifact: produces: Three observations change the operational decision. First, op ticket 42 has two attempt records and two effect observations, but both observations resolve to one provider object. Alerting on “effect rows 1” would be a false positive. The stable provider reference is what proves deduplication. Second, op webhook 77 includes a successful second attempt. A transport only dashboard might close the incident. The two effect references prove that recovery created a second delivery, so the correct state is duplicate and the next task is reconciliation, not another retry. Third, op vm 5 has no duplicate effect in the fixture, but it is still unsafe. The reused key covers two different payload hashes. Waiting until a second resource appears would detect the problem too late; the key conflict is a preventive health failure. The audit has an important limitation: it can only classify the evidence supplied. For op email 3 , no receipt and no lookup path leave the outcome unknowable. The ledger cannot manufacture certainty. Retrying might complete missing work or duplicate completed work, so the bounded response is to surface the ambiguity and ask for authority. Turn the result into a retry boundary Use classification to control the next action, not just color a dashboard: Classification Evidence Safe default healthy One payload identity and exactly one unique effect Stop retrying; verify the intended deliverable duplicate More than one unique effect for one operation Block retries; reconcile or compensate with approval key conflict One key attached to multiple payload hashes Block execution; mint a new operation only after intent is reviewed uncertain No effect proof and no trustworthy absence proof Query again, wait for fresh evidence, or ask a human A time limit is still necessary. Provider idempotency records can expire, lookup indexes can lag, and a destination can be outside the provider's transaction boundary. Store the documented retention and scope beside the key. After that boundary expires, the same request may no longer be safe even if the original code path has not changed. Recovery verification must reach the original outcome. A single provider object can still be wrong: a ticket may exist with the wrong project, or a resource may be created but never become ready. The one effect invariant prevents duplication; a separate outcome contract verifies that the surviving effect is the one the user intended. For an operational health view, report five facts together: 1. the logical operation and payload fingerprint; 2. the attempts and their transport results; 3. the provider's idempotency scope and freshness; 4. the distinct durable effects observed; 5. the authority boundary for retry, compensation, or reconciliation. That makes “retry succeeded” a piece of evidence rather than the verdict. The healthier verdict is “one intended effect exists and has been verified,” or, when evidence is incomplete, “the effect is uncertain; automatic retry is blocked.” Sidewisp is currently in private preview. Its public article system and interactive product demonstration are live, but production agent health collection, runtime adapters, cron management, token cost analytics, and recovery are not generally shipped. The examples above are an operating pattern, not a claim that Sidewisp currently inspects or repairs live agents. Sidewisp is not a replacement runtime, a mandatory gateway, a raw tracing product, an enterprise control plane, or an autonomous fixer. If an effect ledger health rule would help you operate existing agents, consider joining the private preview. Keep execution where it already runs; make retries earn their safety through evidence.