Temporal AI Agents: Separate Durable Execution from Agent Health

Temporal is a strong answer to one hard agent problem: how do you keep a long running execution recoverable when workers crash, processes restart, or an external dependency fails? It is not, by itself, an answer to a different question: is the agent healthy and did it produce the result the user asked for? The safe default is to use Temporal Workflow state as execution evidence, then add four application receipts before assigning a health verdict: 1. a progress receipt showing a meaningful milestone or output delta; 2. a wait receipt naming the owner, deadline, and resume condition; 3. an effect receipt resolving whether a tool side action happened; 4. a deliverable receipt verifying the requested artifact or state. That distinction matters because Temporal's own Workflow Execution documentation defines Running as able to make progress while actively progressing or waiting on something . A green open Workflow therefore cannot distinguish productive work, a legitimate approval wait, or a silent stall. Likewise, a closed Completed Workflow proves that its code reached a completion path; it does not automatically prove that an invoice was sent once, a pull request contains the intended changes, or a report exists at the promised destination. What Temporal proves—and what it does not Temporal's durable execution model gives an AI agent valuable mechanical guarantees. Workflow state persists across failure. Replay checks generated commands against Event History. Activities isolate failure prone calls such as LLM requests, tool use, and external APIs from deterministic orchestration code. The official explanation of dynamic AI agents on Temporal makes this boundary explicit: Workflow orchestration must be deterministic, while LLM decisions and tool results can remain nondeterministic inside Activities. Those properties answer several operational questions: Can recorded orchestration state survive a worker restart? Can the Workflow resume from its recorded history instead of recomputing every prior LLM decision? Is an Activity still retrying, timed out, failed, or complete? Is the Workflow open, paused, cancelled, completed, failed, terminated, or timed out? They do not answer four agent specific questions: Has the plan moved closer to the user's goal, or is the loop merely active? Is a pause expected, owned, and resumable? Did an external side effect occur, especially after a timeout or worker crash? Does the final deliverable exist and satisfy a deterministic acceptance check? This is not a criticism of Temporal. It is a responsibility boundary. The Temporal community AI agent implementation demonstrates an agent loop, tool calls, human confirmation, Signals, state management, and testing inside a Workflow. Its own notes also call out long conversation history, retry visibility, and production storage considerations. Application semantics still belong to the application. Put four receipts above Workflow status A compact receipt can be much smaller than a transcript. It should expose evidence, freshness, and identity without uploading prompts, tool payloads, or secrets. The progress receipt must describe an application milestone, not just a heartbeat timestamp. Temporal documents Activity Heartbeats as a way for a Worker to report liveness and progress, preserve progress details for a retry, and receive cancellation. That transport is useful, but the payload has to carry a meaningful delta: processed record count, verified source set, completed branch IDs, artifact digest, or another task specific invariant. An agent can emit a fresh timestamp forever while repeating the same failed call. The wait receipt prevents the opposite error—paging a healthy human in the loop pause as a stall. Require three fields: owner : the person or system able to resolve the dependency; deadline : when the wait becomes overdue; resumeToken : the Signal, Update, approval ID, or other identity that resumes the same work. Missing any of the three makes the wait operationally incomplete. “Waiting for approval” without an owner is abandoned work. An owner without a deadline can disappear indefinitely. A deadline without a resume identity invites a duplicate or misrouted continuation. The effect receipt is necessary because Activities may be retried. Temporal's Python error handling guidance describes Activities as at least once and recommends idempotency: a Worker can complete an external action and crash before the service records completion. For an agent, the critical states are none , attempted , verified , and unknown . Unknown is not permission to retry. Reconcile the stable operation ID with the destination first. The deliverable receipt closes the gap at the other end. It should bind the Workflow and run identity to a deterministic check: file digest, database version, HTTP resource ID, merged commit, test result, or a structured acceptance verdict. A natural language “done” message is evidence of a claim, not evidence of the outcome. A six case experiment The inspectable fixture used for this article evaluates six runs with one deterministic rule: Workflow state alone would collapse the first four cases into RUNNING and the last two into COMPLETED . The receipts change the operator decision: Case Decisive evidence Safe action Working recent milestone changed leave it alone Waiting owner, deadline, resume token route or wait until deadline Stuck no recent delta and no valid wait investigate, then prepare one bounded recovery Uncertain effect stable operation ID has no destination verdict reconcile; do not retry False success Workflow completed but deliverable is missing reopen the incident Healthy complete completion, effect, and deliverable agree close with evidence The rule is intentionally conservative. It does not use an LLM judge where a deterministic check is available. It preserves uncertain when evidence disagrees. It also avoids “fixing” every pause: a valid wait remains a wait, not a failure. Operate retries and long histories without false green Temporal handles retry mechanics, but the application still owns the retry budget and effect boundary. For each external Activity, carry one stable operation ID across attempts. Record the destination's idempotency verdict when available. Separate transient transport failure from permanent input failure, and stop when the remaining run deadline cannot accommodate another attempt plus reconciliation and deliverable verification. For long Activities, combine three different signals: Activity heartbeat freshness: did the Worker recently communicate? milestone freshness: did useful application state change? attempt and deadline budget: is the current retry still authorized and capable of finishing? A fresh heartbeat with an unchanged milestone may be a loop. A stale heartbeat with a recent destination receipt may be an uncertain reporting failure. An exponential backoff wait can be healthy if its wake time and budget are explicit. No single timestamp deserves a green verdict. History growth is another boundary. The current Workflow Execution limits document hard Event History limits of 51,200 events or 50 MB, with warnings at 10,240 events or 10 MB. Do not turn those dated values into universal constants; check the current documentation and your deployment. The durable pattern is to keep bulky conversation data outside Workflow history when appropriate, retain content minimized identities and invariants, and use Continue As New before history pressure becomes an outage. That creates a practical division of labor: Temporal preserves and resumes orchestration state. The agent application defines milestones, wait ownership, effect reconciliation, and acceptance checks. An operator facing health layer joins both sets of evidence and shows uncertainty rather than inventing a verdict. Keep health separate from orchestration For a Temporal AI agent, durable execution is the foundation, not the final health score. Replay can restore recorded decisions after a crash. Activity retries can recover transient failures. Signals and Updates can carry human decisions. None of those mechanics should be stretched into a claim that the agent is progressing, that a side effect happened exactly once, or that the user's outcome is present. Start with the four receipts. Make them small, fresh, and tied to workflowId , runId , and stable operation identities. Test the six uncomfortable states before production. If your dashboard cannot show waiting , stuck , uncertain , and false success separately, it is hiding the decisions an operator actually needs to make. The limitation is semantic: every workflow must define its own meaningful milestone and deliverable check. A source verification agent and a payment agent cannot share the same acceptance predicate. When no deterministic outcome check exists, label the judgment and its confidence; do not silently convert it into fact. Sidewisp is currently in private preview. Its product direction is an AI agent health layer, but a production Temporal adapter and live agent health collection are not presented here as shipped capabilities. The useful near term move is independent of any product: keep Temporal's durability evidence, add the four application receipts, and require them to agree before calling an agent healthy.