AI Agent Observability for Rate Limits: Measure Retry-After Debt

An HTTP 429 does not by itself mean an AI agent is stuck. Treat the run as waiting only while four proofs hold: the provider's not before boundary is known, a durable retry is scheduled on or after that boundary, retry authority remains, and the expected outcome still has deadline headroom. If any proof fails, the operator needs a different diagnosis—not another generic retry. This distinction matters because the same quiet process can be healthy backpressure, an early retry loop, a missed wake, or a task that can no longer finish on time. Request counts and process activity cannot tell those states apart. Short answer: wait only while four proofs hold Start with one event contract for every throttled call: Then evaluate the proofs in this order: 1. Boundary: can the client normalize the provider signal to retry not before ? 2. Wake: is there a durable scheduled retry at or after that instant? 3. Authority: does the run still have an allowed attempt, time, and cost budget? 4. Outcome: does outcome deadline retry not before leave enough time to complete and verify the intended work? A run is not healthy merely because it sleeps until the correct second. Suppose the provider asks for a 15 minute wait but the deliverable is due in 10 minutes. The client can obey the protocol perfectly while the task is already operationally lost. Escalate that conflict instead of showing a green wait. Define a local diagnostic metric: retry after debt ms is introduced here as an operating measure, not an HTTP field, provider bill, or universal SLO. It separates time deliberately surrendered to provider backpressure from model execution, tool work, scheduler delay, and result verification. Trend it per run and quota scope; do not combine unrelated tenants or resources into one misleading total. Record the provider boundary before judging the agent RFC 6585 defines HTTP 429 as “Too Many Requests.” A response may include Retry After , but the standard deliberately does not define whether the provider counts by credential, resource, server, or another scope. Your health event therefore needs both the response and the best available quota scope key. A global “provider throttled” label is too coarse when only one project or endpoint is constrained. HTTP Semantics defines Retry After as either an HTTP date or a non negative delay in seconds. Preserve the raw value for investigation, but normalize it immediately: For an HTTP date, record the client clock offset if you can. For a missing or invalid field, set the boundary to unknown. A policy may then choose bounded exponential backoff, but observability should say uncertain boundary ; it should not invent provider permission to retry. The local schedule needs its own receipt. Store scheduled retry at , scheduler job ID, attempt number, and the last confirmed scheduler heartbeat. When a retry actually starts, emit retry started at ; when the provider responds, emit retry finished at and the new status. That makes two opposite failures visible: Early retry loop: retry started at < retry not before . The client is adding pressure before the declared boundary. Missed wake: current time exceeds scheduled retry at + wake grace , but no retry start receipt exists. The absence of traffic is healthy in the first waiting window and unhealthy after the wake grace. Silence alone is not a state. A concrete production implementation reinforces the need for bounded authority. The AWS SDK retry guide separates throttling from transient failures, uses exponential backoff with jitter, and stops when maximum attempts or retry quota are exhausted. The exact AWS delays are not a universal agent policy. The reusable lesson is to expose classification, backoff, and stop conditions instead of hiding them inside a client library. Run the six case audit The inspectable fixture for this article fixes “now” at 2026 07 26T18:42:00Z and gives every run one 429 snapshot. The audit applies a 30 second wake grace and checks recovery before failure states, then budget, deadline, early retry, missed wake, and valid waiting. The six NDJSON rows produce six different results: Healthy wait — waiting backpressure : a 60 second boundary, aligned wake, three attempts, and nine minutes of headroom. Early loop — early retry loop : a retry starts 105 seconds before the provider boundary. Missed wake — stuck missed wake : the scheduled time and grace pass without a retry receipt. Deadline blocked — deadline exhausted : the not before instant lands five minutes after the outcome deadline. Budget exhausted — retry budget exhausted : the boundary is short, but no authorized attempt remains. Recovered — recovered : a post boundary retry returns 200 and an outcome receipt follows. The measured total is 1,230,000 milliseconds of retry after debt: 20.5 minutes across the six snapshots. That number is useful because it is inspectable, but it is not automatically bad. Sixty seconds in the healthy wait is intentional. Nine hundred seconds in the deadline blocked run is decisive because the remaining headroom is negative. Interpret debt beside outcomes, not as a standalone score. The recovery row also prevents a common false success. A 200 response proves that one retry completed; it does not prove that the agent produced the requested file, sent the approved message, updated the record, or passed validation. Close the incident only when a deterministic outcome receipt matches the original run and expected deliverable. Turn each state into one bounded action Use one action per diagnosis: For waiting backpressure , leave the run alone and verify that the durable wake still exists. For early retry loop , pause that retry path, preserve the last provider boundary, and inspect whether multiple retry layers are multiplying requests. For stuck missed wake , run one scheduler check. Recreate or trigger a retry only within the original authority and attempt budget. For deadline exhausted , notify the owner that the current outcome cannot meet its deadline. Do not conceal the conflict with a longer timeout. For retry budget exhausted , stop and surface the final provider evidence. A larger budget is a human policy decision. For recovered , verify the intended outcome before clearing the issue. For an unknown boundary or quota scope, mark the state uncertain and collect evidence; do not guess that the agent is either healthy or broken. Keep the limitation close to the decision. Providers may omit Retry After , expose several overlapping quotas, or throttle behind an intermediary. Clocks can drift. SDKs may retry internally before the agent runtime sees an error. Instrument the lowest layer that can expose attempt receipts, then correlate upward by run and attempt IDs. Never log bearer tokens, prompts, response bodies, or secret quota keys just to diagnose timing. Sidewisp is currently in private preview. The public site and article system are live, but production agent health collection, runtime adapters, cron management, token cost analytics, and recovery are not generally shipped. Sidewisp is not a replacement runtime, mandatory gateway, raw tracing product, enterprise control plane, or autonomous fixer. The practical reason for joining the private preview is to help shape health evidence like provider boundaries, durable wakes, retry budgets, and verified outcomes—not to obtain a monitoring capability that is already generally available. The operating rule is narrow: honor provider backpressure, but do not confuse compliant waiting with healthy progress. A rate limited run stays healthy only while its boundary, wake, authority, and outcome deadline agree; after the retry, only the intended outcome closes the loop.