AI Agent Timeout Health: Build a Deadline Budget

An AI agent needs one absolute run deadline , not a fresh timeout for every model call and tool. Before each expensive stage, calculate: Continue only when the remaining budget covers the required budget and useful progress is still changing. If the run has 75 seconds left but needs 90 seconds of work plus 15 seconds to cancel, reconcile side effects, and verify the outcome, it is already in timeout risk even while spans and heartbeats remain fresh. This is the practical role of deadline aware AI agent observability: show whether the current run can still deliver a verified result inside its time boundary. A timeout on one call is merely a local limit. It does not prove that child work stopped, that a retry is safe, or that the requested result exists. Give the whole run one shrinking time budget gRPC defines a deadline as the point after which a client is unwilling to wait. Its documentation recommends explicit, realistic deadlines because no deadline can otherwise leave a client waiting indefinitely. It also distinguishes a deadline from a timeout: the deadline is an absolute point in time, while a timeout is a maximum duration that can be converted to a deadline when the call begins. That distinction matters in an agent run. Consider a job with a five minute user deadline: 1. Planning consumes 40 seconds. 2. A model call consumes 55 seconds. 3. A tool waits in a queue for 70 seconds. 4. The agent starts another tool with its usual two minute timeout. The fourth stage may be locally configured correctly, but only 135 seconds remain before accounting for result validation and cleanup. Starting a fresh two minute call has silently allocated almost the entire residual budget. Starting another fresh timeout after that would extend work beyond the promise made to the user. Carry one deadlineAt value through the run. At every child boundary, derive a shorter local timeout from the remaining budget. Never reset the original deadline. gRPC’s propagation guidance describes the same reliability principle for RPC trees: pass the caller’s deadline downstream and deduct time already elapsed, rather than granting each child a new full interval. Keep the health record small and inspectable: Field What it establishes What it cannot establish deadlineAt The latest acceptable end of the run That child work will honor cancellation estimatedRemainingSeconds Current estimate for useful work left That the estimate covers an unseen branch cleanupMarginSeconds Reserved time for cancellation and verification That cleanup is bounded in every provider lastProgressAt Freshness of task specific movement That activity produced the right result progressChanged An inspectable milestone or fingerprint changed That the final outcome is correct terminal The runtime ended its path That the deliverable exists outcomeVerified An external acceptance check passed That every unstated expectation was met The progress field should reflect the job, not generic traffic. A canonical artifact digest, test result, destination object ID, row count, or monotonic milestone can show useful movement. Token counts, log volume, and tool call totals show activity but can increase during a loop. Test the deadline rule against six boundary cases The accompanying artifact fixes the observation time and runs six synthetic snapshots through a deterministic classifier: The run produced: fresh build has 180 seconds left. Its estimated remaining work is 90 seconds, its cleanup margin is 15 seconds, and its task fingerprint changed 30 seconds ago. The required budget is 105 seconds, so the run remains feasible and the classifier returns working . slow export also has recent changed progress, but only 75 seconds remain. The same 90 second work estimate plus 15 second margin requires 105 seconds. Activity is healthy; feasibility is not. The correct verdict is timeout risk , not working and not yet deadline exceeded . quiet retry demonstrates a different failure. It has 300 seconds left and needs only 150, so its deadline math passes. However, its useful progress evidence is 1,200 seconds old and unchanged beyond the fixture’s 600 second stall window. It returns stalled . Adding more time would not address the evidence that the run is repeating without movement. legacy task has no deadline or remaining work estimate. Fresh activity cannot repair missing timing evidence, so it stays uncertain . expired call is ten seconds beyond its absolute deadline and returns deadline exceeded . published report becomes complete only because terminal execution is paired with an independent outcome check. The decision order is deliberate: 1. Accept complete only with terminal execution and verified outcome. 2. Return deadline exceeded when the absolute deadline has passed. 3. Return timeout risk when remaining time cannot cover work plus margin. 4. Return stalled when time remains but progress is stale and unchanged. 5. Return working only when the run is feasible and evidence is moving. 6. Keep missing or contradictory timing evidence uncertain . This order allows a progressing run to be unhealthy because it cannot finish in time, while keeping a stalled run separate from a time budget failure. The fixture is an inspectable decision test, not evidence that these states occur with equal frequency. Its 600 second stall window and timing estimates are example policy values. A real adapter must derive them from the task class and observed duration distribution. Propagate cancellation as well as the deadline A deadline that stops the parent from waiting is not proof that child work stopped. gRPC explicitly notes that server applications are responsible for stopping activity they spawned after cancellation. That boundary is especially important for agents: a timed out tool may still be exporting data, writing a file, charging an account, or holding a lock after the orchestrator has moved on. Google’s SRE guidance on cascading failures describes missed RPC deadlines as wasted work that can invite retries and further overload. It also explains that cancelling other work in a call tree prevents resources being spent on a result that can no longer be delivered. An agent operator should apply the same principle without assuming that every tool supports cooperative cancellation. For each child stage: pass the absolute deadline when the protocol supports it; otherwise derive childTimeout = deadlineAt now reservedMargin ; refuse to start when the derived timeout is non positive or implausibly short; propagate an abort or cancellation signal; record whether the child acknowledged cancellation; reconcile externally visible effects before any retry; reserve enough time for the independent outcome check. A compact event can stay free of prompts and responses: An opaque or keyed reference is safer than a raw user identifier or filesystem path. Do not send prompts, responses, credentials, tool payloads, or sensitive destination names merely to calculate deadline health. Cancellation also needs an outcome boundary. If a write request times out after leaving the host, the remote service may have committed it before the response disappeared. Retry only after checking an idempotency key or querying the destination. A second attempt made inside the remaining budget can still be wrong. Estimate the budget without pretending it is certain The reasonable default is to estimate each task class from observed high percentile duration, then add explicit margins for queue delay, cancellation, reconciliation, and outcome verification. Update the estimate when the planned branch changes. A file inspection and a repository wide test suite should not share one generic timeout. Keep the estimate version in the evidence so operators can explain a verdict. Alert on transitions such as working → timeout risk instead of every decrement of the clock. If the collector clock is ahead of the runtime, negative ages can invert the decision; record both observation time and source time, reject impossible values, and prefer monotonic durations within one process. OpenTelemetry’s AI agent observability overview argues for interoperable traces, metrics, and logs through emerging semantic conventions. Those signals are useful inputs, but a standard span does not know the user’s promised finish time, which artifact counts as progress, or how long outcome verification needs. Those remain task level contracts. The model also has hard limits. Duration estimates fail on new task shapes. Providers may ignore cancellation. A child may finish after the parent deadline. Clock skew, queueing, rate limits, or an unobserved branch can consume the margin. Deadline health is therefore evidence with freshness and confidence, not a guarantee. The reusable rule is narrow: propagate one absolute run deadline, spend from it before every stage, reserve cleanup and verification time, and classify stale progress separately from insufficient time. Completion still requires the intended result, not merely a stopped clock. Sidewisp is currently in private preview. Its public site and article library are live, but production agent health collection, runtime adapters, deadline monitoring, and recovery are not generally shipped. Sidewisp is intended to work alongside existing runtimes and keep human authority visible. If end to end deadline health is a failure you need surfaced, you can join early access without treating this article as a claim of deployed monitoring.