LLM Observability for Retry Storms: Cost per Verified Outcome

LLM observability should count tokens, latency, errors, and model calls. For an agent that can retry work, that is only the numerator. The operational denominator is the number of intended outcomes verified at the destination. A useful cost signal is therefore: Keep retry cost, retry owner, and outcome status under one stable run id . Do not divide spend by successful API responses or by the agent's own “complete” message. Both can look healthy while a workflow repeats, a deliverable is absent, or two layers quietly retry the same failure. This guide applies that rule to a fixed eight run experiment. The steady cohort costs $0.012 per verified outcome. The retry storm cohort appears to cost $0.041 per declared completion, but its destination check proves only one result, so the actual figure is $0.123—10.3 times the steady cohort. The dollar amounts are synthetic; the accounting error is real and reproducible. Keep the normal LLM observability layer The reasonable default is still model call telemetry. Record request duration, input and output tokens, error class, provider, model, operation, and trace correlation. Those signals tell you whether a provider slowed down, a context expanded, a model changed, or a call failed. The current OpenTelemetry GenAI metric conventions make that baseline concrete. At the pinned July 24, 2026 commit, they define gen ai.client.token.usage and gen ai.client.operation.duration . They also define agent level inference call and tool call counts. The document marks the conventions Development , so pin the version you implement and expect fields to move. Token usage is not automatically cost. A provider may return billable token counts, a gateway may calculate an estimate, and an invoice may later reconcile the amount. Keep the provenance beside the value: Use an opaque run identifier. Prompts, responses, credentials, tool arguments, customer content, and absolute paths do not belong in a cost dimension. The protected trace can remain available for authorized investigation; the aggregate only needs enough information to locate the attempt and explain its accounting. Per call graphs remain valuable. They simply answer a different question. A falling cost per model response can coexist with rising attempts per workflow. A successful retry can repair a transient provider error while hiding that the same task was also retried by a queue and then by the agent. Model telemetry describes the calls. Run accounting describes the promise. Make the verified outcome the denominator Define the outcome before execution. “The model returned text” is a call result. “The pull request has the expected commit,” “the report exists under the agreed key,” or “the sitemap contains the published URL” is an outcome. The minimal run record needs four states: State Meaning Cost treatment Health treatment verified A destination native predicate passed Include cost and increment the denominator Complete missing The agent declared completion but the predicate failed Include cost; do not increment the denominator False success waiting A named dependency or human decision is outstanding Include cost; do not call it success or failure yet Route to the dependency owner unavailable The verifier did not run or its evidence is stale Include known cost; leave the ratio unknown if no valid denominator exists Investigate evidence coverage This distinction prevents a convenient but destructive shortcut. If a person has not approved a change, the agent is waiting; repeatedly running the model does not create authority. If the verifier is offline, treating its absence as failure can trigger duplicate side effects. If the agent says “done” but the destination is empty, treating the declaration as success rewards false completion. Join the outcome record to attempts, rather than copying the deliverable into the observability store: The verifier should be deterministic where possible. Check a file hash, database row, API field, test result, or destination status. A qualitative evaluator can supply evidence when the result cannot be expressed as a predicate, but its version, calibration, and uncertainty belong beside the score. Reproduce the retry cost gap The accompanying experiment uses eight synthetic runs: four steady and four in a retry storm. Each run contains attempt level token and cost fields plus one final outcome state. The storm includes one verified outcome, two false success declarations, and one legitimate approval wait. Save one NDJSON object per run. This abbreviated pair shows the shape: Aggregate every attempt under its cohort, then calculate: The complete fixture and audit retained with this publication produce: Three observations change the operating decision. First, the storm's cost per declared completion understates cost per verified outcome by 3×. The agent's declaration is a poor billing denominator. Second, attempt amplification rises from 1.25 to 3.00. A model call dashboard can show twelve individually normal calls without showing that they belong to only four promises. Third, 62.6% of storm spend occurs after the initial attempts, and two different layers own those retries. The issue is not merely an expensive model. It is an unbounded control path. This experiment does not estimate a production failure rate. Its prices and cases are constructed to test the accounting rule. Run the same calculation on your own billing or provider derived costs, retain the source and price version, and compare like for like workflows over time. Give one layer the retry budget Retries are often correct. A throttled request or transient network error may succeed after a delay. The failure begins when every layer independently decides it owns recovery. The OpenAI rate limit guidance recommends random exponential backoff and warns that unsuccessful requests still contribute to the per minute limit. Continuous resending therefore consumes the capacity needed for recovery. The current AWS SDK retry reference documents the same control principles in a broader API setting: bounded maximum attempts, exponential backoff with jitter, and a retry quota token bucket that stops retries when its budget is depleted. Those sources do not prescribe one universal agent policy. They support a safer contract: 1. Pick one retry owner for an operation—usually the lowest layer that can classify the transient error and preserve idempotency. 2. Count the initial request and every retry against one run level attempt and cost budget. 3. Propagate attempt metadata upward so a workflow runner does not mistake an SDK's final error for a first failure. 4. Make non retryable states explicit: denied permission, invalid input, missing authority, and failed outcome verification need routing or investigation, not blind repetition. 5. Stop when the time, attempt, or cost budget is exhausted. Return a visible state with the last evidence. 6. Verify the destination after a retry. A command that returned successfully is not the promised result. Timeouts need special care. A client timeout does not prove that the remote side did nothing. Before retrying a side effecting tool, use an idempotency key or query the destination. Otherwise, an observability system may correctly report the second attempt while the business system receives two invoices, messages, or publications. Alert on regression, then inspect the outcome Do not page on one retry. Start with workflow specific baselines and require persistence. A useful first warning can combine three conditions: Tune those values to the workflow. A batch process with cheap, idempotent fan out may tolerate more attempts. A payment, publication, or customer message workflow may permit fewer. Separate provider throttling from tool failure and from an absent destination result. They have different owners and different safe actions. The alert should name the affected promise, total attempts, retry owners, cost provenance, verifier result and freshness, and the next bounded action. A useful message says: “Report publication used 12 attempts across the SDK and workflow runner; retry spend is 63%; one of four outcomes is verified; inspect retry ownership and the sitemap verifier.” It should not say only “token cost high.” There are two important limits. Cost data can be delayed, estimated, or incomplete, so show coverage and do not fabricate zeroes. Outcome checks can also fail independently, so unavailable must remain distinct from missing . A legitimate waiting run stays out of the verified denominator without being labeled stuck until its dependency or deadline changes. Sidewisp's product direction includes cost as one health signal alongside availability, execution, memory, tools, and outcomes. It is intended to work beside existing runtimes, not become a mandatory model gateway or an autonomous fixer. Sidewisp is currently in private preview. The public site and article library are live; production monitoring adapters, token cost analytics, and recovery execution are not generally shipped. Join the private preview if you want to help shape how retry evidence, cost, and verified outcomes should meet while humans retain authority.