AI Agent Observability for Split-Brain Runs: Fence Stale Workers

AI agent observability needs an ownership signal, not just more traces, when a failed over worker can resume. Give every acquisition of a durable workflow a monotonically increasing owner epoch . Put that epoch on progress events and external effect attempts. At the destination, accept an effect only when its epoch is still current. This separates three facts that are easy to blur: a worker is alive; a worker is making activity; a worker is still authorized to change the outside world. A heartbeat supports the first claim. A checkpoint may support the second. Neither proves the third after another worker has taken over. Two executions can each look locally coherent while both write a release index, send a customer message, update a ticket, or publish the same artifact. The reasonable default is a lease for takeover plus a fence at every irreversible destination. Use the lease to decide when another worker may become owner. Use the fence to stop the displaced worker if it wakes up late. Observe both paths, because the lease service can be healthy while a destination ignores the ownership token. A current lease is not an effect fence Kubernetes Lease documentation gives a useful concrete model. Lease objects support node heartbeats and component leader election. A kubelet updates spec.renewTime , and the control plane uses that timestamp when deciding node availability. The Lease API also exposes holder identity, lease duration, and transition information. Those fields answer freshness and election questions. They do not make an old process disappear. A worker can pause during a network problem, VM suspension, long garbage collection stop, or blocked tool call. The lease can expire, a replacement can acquire ownership, and the old process can then resume from local state. The boundary is not hypothetical wording. The Kubernetes client go leader election package explicitly says its implementation does not guarantee that only one client is acting as leader—also called fencing. The package is designed around coordinated election and clock skew tolerance, not a universal guarantee that every downstream system rejects the old leader. Evidence What it supports What it does not prove recent heartbeat the process or observer was recently reachable the process still owns the workflow current lease holder the coordination store selected this owner the previous owner cannot reach a tool trace with successful spans one execution path completed recorded steps no competing execution produced the same effect increasing checkpoint this worker changed local state its changes are authorized or useful sink acceptance with current epoch this destination accepted the current owner every other destination enforced the same rule Call the condition split brain execution only when ownership evidence conflicts: a higher epoch has been acquired, yet a lower epoch still reports activity or attempts an effect. Do not label an ordinary handover as an incident. The old worker may have stopped cleanly, and the new worker may be the only actor after takeover. That distinction prevents two bad alerts. “Two workers existed” is too broad; rolling replacement can be healthy. “Both workers emitted logs” is also too broad; buffered evidence can arrive late. The useful question is whether an event occurred after the higher epoch became authoritative, using the coordination store's ordered transition or another authoritative sequence—not whichever machine clock looks newer. Put the owner epoch on the effect An observable ownership record can stay compact. It should identify the durable workflow, the acquisition, the worker, the effect, and the observation order: workflow key is the unit that must have one effect owner. It is not necessarily a trace ID or process ID. A scheduled export could use tenant/export/date ; an inbox agent could use the source message ID; a publishing workflow could use locale plus article slug. owner epoch is a monotonically increasing token allocated by the authoritative coordination store. A timestamp from the worker is not a safe substitute. Clocks can move, and two hosts can disagree. A random run ID is useful for joining evidence but has no ordering relation. The audit needs to know that epoch 18 displaced epoch 17. effect key names the externally visible result closely enough to detect two owners targeting the same outcome. “Tool call 44” is weak because two runs will choose different call IDs. “Release index for July 26” or “reply to source message 8f2…” describes the thing that must not happen twice. Record at least these event types: lease acquired : authoritative transition to a higher epoch; progress : a meaningful checkpoint, still separated from authority; effect attempted : the worker is about to cross a side effect boundary; effect committed : the destination confirms the effect; outcome verified : an independent predicate confirms the intended result. The detector's state is simple. For each workflow key , retain the highest acquired epoch. Evidence from a lower epoch after that acquisition is stale. A stale progress event is diagnostic: an old process is still active. A stale effect attempted event is an actionable near miss if the destination rejects it. A stale effect committed event is a correctness incident because the fence failed or did not exist. Do not upgrade stale activity directly into a claim that damage occurred. Activity and effect are different. The old worker might finish a local calculation, write a disposable cache, or shut itself down. Severity should rise when the stale epoch reaches a destination, and rise again when two epochs commit the same effect key . Replay one unsafe takeover and one clean handover The accompanying fixture has sixteen ordered events across three workflows. export ledger starts with worker alpha at epoch 17. worker beta then acquires epoch 18. Alpha resumes, emits a progress checkpoint, attempts the shared release index effect, and—representing an unsafe destination—commits it. Beta commits the same effect at epoch 18. report index provides the control case. Epoch 5 commits one shard, epoch 6 later takes over and commits a different shard, and no lower epoch event occurs after the transition. checkout sync stays on one owner. Run the audit: The deterministic result is: PASS means the acceptance test found every planted condition. It does not mean the unsafe export ledger workflow was healthy. Sequence 6 is stale activity from epoch 17 after epoch 18 exists. Sequence 7 is the stale attempt. Sequence 8 is the unsafe stale commit. When sequence 9 commits the same effect from epoch 18, the audit can show both owners and both source events instead of merely reporting a duplicate count. The experiment yields four practical observations. First, acquisition count is not an error metric. Both export ledger and report index change owner. Only the former has lower epoch evidence after takeover. Second, a checkpoint can prove that a process is doing work while simultaneously proving that the work is unauthorized. “Rows processed increased from 400 to 600” is activity, not permission. Third, duplicate detection becomes explainable when it retains the epochs and transition order. An operator can see whether the duplicate came from a client retry by one owner or from two owners acting across failover. Fourth, this is an audit of evidence, not a distributed lock proof. The fixture uses an authoritative sequence so the decision rule is inspectable. Real systems must define where epochs are allocated, how that allocation is made durable, and which destinations enforce it atomically. Enforce the fence at each destination Logging owner epoch only diagnoses a stale writer after the fact. Prevention must live at the system that owns the effect. In database backed work, that can be one transaction: lock or compare the workflow's current epoch, reject a lower value, then write the effect and its idempotency key before committing. The comparison and effect must share an atomic boundary. Checking the epoch, releasing the lock, and then calling an external API leaves a race between the check and the effect. Kafka documents one concrete version of the idea. ProducerFencedException indicates that another producer with the same transactional.id has started; the latest instance fences previous instances so they can no longer make transactional requests. This does not turn Kafka's mechanism into a universal agent protocol. It demonstrates the property to ask of a destination: can a newer owner invalidate an older one at the point of commit? Many agent tools cannot compare an epoch. Email APIs, ticket systems, shell commands, and SaaS mutations often accept a request without consulting your lease store. Use the strongest boundary that destination supports: 1. Pass a fence token and require an atomic compare when you control the sink. 2. Use a destination enforced idempotency key when duplicate effects are equivalent. 3. Stage output under the epoch, then let one current owner transaction promote it. 4. Put a transactional outbox between the agent and the external API. 5. When none is possible, reconcile by effect key , expose uncertainty, and require human review for costly or irreversible repeats. Idempotency and fencing solve related but different problems. An idempotency key can collapse repeated requests for the same effect. A fence rejects all later requests from an old owner, including a different effect key that the old plan should no longer produce. For critical workflows, use both. Availability is the trade off. If the coordination store cannot allocate or confirm a current epoch, rejecting effects may pause useful work. That is preferable for money movement, publication, destructive changes, or customer communication. A read only research task may instead continue locally and delay only the commit. Set the boundary by effect risk, not by a blanket desire to keep every agent busy. Turn the conflict into an operational health issue A stale owner finding should include the workflow, displaced and current workers, both epochs, the authoritative transition, the latest stale event, affected effect keys, evidence freshness, and whether the sink rejected or committed the request. Confidence is high only when the acquisition order and effect acknowledgement come from authoritative sources. The safest response depends on what happened: stale activity with no attempted effect: stop or quarantine the old worker if that action is authorized, then verify it emits no further events; rejected stale attempt: preserve the rejection receipt, inspect why the worker missed lease loss, and verify the current owner still progresses; stale commit without a competing commit: freeze further effects, inspect the result, and decide whether compensation is safe; two committed epochs for one effect: treat the outcome as uncertain until an external predicate or human verifies the durable result. Do not automatically “fix” a split brain condition by retrying the current owner. That can create a third effect. Clear the issue only after the stale worker is fenced and the intended outcome—not merely a command exit—is verified. Sidewisp's product direction is a health layer around existing agent runtimes, focused on evidence, useful progress, outcomes, and explicit approval boundaries. It is not a replacement runtime, mandatory gateway, raw tracing product, enterprise control plane, or autonomous fixer. Sidewisp is currently in private preview. The public site and article system are live, while production agent health collection, runtime adapters, cron management, token cost analytics, and recovery are not generally shipped. If stale owner evidence is one of the failure modes you need to operate, you can join the private preview and describe the runtime and effect boundaries involved. Primary references Kubernetes: Leases — node heartbeat freshness, leader election, and Lease objects; reviewed July 26, 2026. Kubernetes client go: leader election — implementation scope and the explicit absence of a one active client fencing guarantee; reviewed July 26, 2026. Apache Kafka: ProducerFencedException — latest transactional producer fencing previous instances; reviewed July 26, 2026.