Agent Observability for Queue Starvation: Track Runnable Age

Agent observability should not call a queue unhealthy merely because it is long. It should ask how long the oldest runnable task has waited, whether any worker is reachable, whether a slot is free, and whether verified outcomes are still arriving. Those facts distinguish a legitimate dependency from insufficient capacity, a dead worker, a poisoned task, or a dispatcher that has stopped assigning work. The practical default is to record eligibleAt for each task and alert when now eligibleAt exceeds that task class's start objective. Do not start the clock while a declared dependency or retry delay is still valid. Queue depth remains useful context, but runnable age is the decision signal. Queue depth is context, not a diagnosis A count compresses unlike states. Ten tasks may be waiting for a human approval, ready to start, already leased to workers, delayed by backoff, or repeatedly failing. Treating all ten as one backlog makes a busy queue look broken and can hide one old task behind a small count. The queue services themselves expose the limitation. Amazon SQS publishes both ApproximateNumberOfMessagesVisible and ApproximateAgeOfOldestMessage , and labels many values approximate because of its distributed architecture. Google Pub/Sub is more explicit in its monitoring guidance: the absolute number of unacknowledged messages is not necessarily meaningful, while a small steady backlog with a steadily growing oldest message age can indicate stuck messages. For agent work, raw message age is still too coarse. A task enqueued at 09:00 but blocked on an approval until 10:00 should not spend an hour of its start budget before it is eligible. Define: Leave eligibleAt absent while a bounded dependency is open. Record the dependency kind, owner, and deadline separately. If neither eligibility evidence nor a valid dependency exists, return uncertain ; do not convert missing data into a healthy zero. Build an eligibility ledger A minimal record can stay content free: Field Operational question taskId Which safe opaque task identity is affected? eligibleAt When could a worker legitimately start it? dependency Who or what owns the wait, and until when? deliveryAttempts Has the task exhausted its bounded retry policy? workerHeartbeatAt Is at least one compatible worker reachable? slots and active Is capacity occupied or available? lastVerifiedProgressAt Are completed outcomes still moving? This is deliberately smaller than a trace. The OpenAI Agents SDK tracing documentation describes generations, function calls, guardrails, handoffs, and custom events. Those records help explain execution, but they do not say when a queued task became eligible or whether the intended result was observed. Join detailed traces to the ledger by a safe run identifier; do not make trace activity stand in for queue progress. Use a decision order that preserves the cause: 1. Missing eligibility and dependency evidence is uncertain . 2. No runnable tasks plus a valid dependency lease is waiting . 3. Runnable age inside the start objective is healthy . 4. An old oldest task beyond its delivery budget is poisoned head . 5. Old runnable work plus stale worker heartbeats is worker unreachable . 6. Old runnable work, all slots occupied, and fresh verified progress is capacity bound . 7. Old runnable work plus a fresh worker and a free slot is dispatcher stuck . Order matters. If a task has already exhausted its attempt budget, adding workers is not the first repair. If no worker is reachable, blaming dispatch is premature. If all slots are busy and outcomes are still arriving, the system is slow against its objective but not motionless. Replay seven queue states The accompanying artifact freezes the observation time at 2026 07 26T04:50:00Z , sets a 120 second worker freshness window and a 300 second start objective, then evaluates seven synthetic queues. The run produced: dispatcher gap is the decisive case. Its oldest runnable task has waited 900 seconds, the worker heartbeat is only 20 seconds old, and both slots are free. More capacity would not help; assignment or routing evidence should be inspected. By contrast, all slots busy has an 840 second runnable age, no free slot, and a verified outcome 80 seconds ago. It is capacity bound under this fixture's objective. bounded dependency was enqueued earlier than either case, yet it is waiting : the approval has a named owner and a future deadline, so there is no runnable clock to violate. silent worker keeps old ready work separate from reachability failure. poisoned oldest prevents a fifth failed delivery from disappearing inside a normal looking queue. missing eligibility stays uncertain. The experiment demonstrates the decision rule, not prevalence. One synthetic case per state cannot establish production thresholds, and it does not model priority inversion, partitioned queues, clock skew, or task affinity constraints. Pair age with capacity and outcome movement Runnable age becomes actionable only beside capacity and progress. An old task with every compatible slot occupied suggests a scaling or workload shaping decision. The same age with an open slot points at dispatch, routing, task affinity, or a lost lease. Avoid three tempting shortcuts: Do not average away starvation. A low mean start delay can coexist with one task that never runs. Track the oldest runnable age and a percentile by task class. Do not mix blocked and runnable work. Keep dependency age visible, but exclude it from the start objective until the dependency resolves or its lease expires. Do not infer progress from leases or tool calls. Capacity is genuinely moving only when a task specific artifact, acceptance check, or destination receipt changes. Choose the start objective from the job's promise. An interactive coding task, a scheduled report, and a nightly reconciliation should not share 300 seconds because the fixture does. Measure normal eligibility to start time, set a reviewable objective with a buffer, and version it. Partition by compatible worker pool or task class so one unrelated queue cannot mask starvation. When a dependency deadline passes, do not silently extend it. Recompute eligibility from the evidence you have and notify the named owner. When a worker heartbeat is stale, verify connectivity before retrying jobs that might still be running elsewhere. When a poisoned task reaches its delivery budget, quarantine or request review rather than allowing it to monopolize the head of the queue. Keep diagnosis separate from intervention Runnable age tells you that an operational promise is late; the joined facts suggest why. They do not authorize an automatic repair. A dispatcher stuck verdict can prepare a bounded dispatch check. capacity bound can open a capacity review. worker unreachable can request a reachability check. None of those states alone permits restarting a host, duplicating a side effect, changing credentials, or spending beyond a retry limit. After any approved action, require new evidence: the task received a lease, a progress fingerprint changed, or the expected outcome was independently verified. A command that returned zero is activity, not recovery. There are important boundaries. Provider age metrics may be approximate. Clock skew can create impossible negative ages, so compare collector time and producer time before using the result. Priority queues may legitimately let low priority work age; expose the policy rather than calling it healthy by accident. A task can also hold a free looking slot through an unobserved lease, so missing lease data should lower confidence. The resolved rule is narrow: start the clock when work is actually runnable, alert on the oldest runnable task against a task specific objective, then use worker freshness, free slots, retry budget, and verified outcome movement to classify the cause. Sidewisp is currently in private preview. Its public site and article library are live, but production agent health collection, runtime adapters, and recovery are not generally shipped. Sidewisp is intended to work alongside existing runtimes, not replace them or act as an autonomous fixer. If runnable age evidence would make your agent operations easier to judge, you can join early access while treating the product as a preview rather than deployed monitoring.