Helicone LLM Observability: Prove Every Model Route Is Covered

Seeing requests in Helicone answers an important question: some traffic is observable . It does not answer whether every model call route is represented, whether one provider attempt was recorded twice, or whether the agent produced its promised result. The practical fix is to define the denominator outside the dashboard. Keep a small route manifest for every production path that can call a model. For each real provider attempt, expect exactly one Helicone observation through the declared proxy or async method. Then classify the work state and verify the destination separately. That order matters. A dashboard can be internally correct while an emergency fallback bypasses it. It can also overcount when the same call goes through the proxy and an async wrapper. Neither request count is an agent health verdict. Start with the routes that can actually send work Helicone documents two integration shapes with a real architectural trade off. Its proxy versus async comparison says the proxy is the request gatekeeper: the application changes its base URL, Helicone forwards the call, and gateway features such as caching, retries, and rate limiting can run on that path. Async logging stays off the critical path, so a Helicone or logging network problem need not interrupt the application, but it does not provide the same gateway feature set. That is a route level choice, not a one time account setting. A typical agent service may contain all of these: Route Example caller Intended observation mode Operational reason chat primary interactive API gateway routing and retry policy live on the request path batch summarizer background worker async logging must not extend the batch critical path emergency fallback direct provider client gateway a fallback is useful only if it remains visible nightly evaluator scheduled Python job async evaluation traffic should be separated from user work The dangerous row is not necessarily the one with errors. It is the route that exists in code or configuration but has no declared observation contract. Create one privacy minimal record per provider attempt: work id identifies the accepted unit of work. provider attempt id identifies one real call, including a retry. route says which application path produced it. None of those fields needs a prompt, response, API key, email address, or absolute host path. Helicone exposes request, custom property, user, and session identifiers in its header directory. Use the minimum correlation metadata your application can validate. Do not put secrets or arbitrary user content into a custom property merely because the field accepts a string. Sessions solve a different problem. Helicone's Sessions documentation groups logged LLM calls, vector queries, tool calls, and other requests with application supplied IDs and paths. That helps reconstruct a flow, but it cannot discover a provider call that never reached a logging path. The same documentation warns that reusing one session ID mixes unrelated work. A session is therefore useful context, not the coverage denominator. Reconcile provider attempts before reading totals The audit rule is deliberately strict: 1. Enumerate every provider attempt the application says occurred. 2. Find observations with the same stable attempt identity. 3. Require exactly one observation through the route's declared mode. 4. Only then interpret work state and outcome evidence. The companion fixture contains eight content free cases. Run it with: The deterministic result is: Two cases deserve attention because the model call itself completed and the destination existed. In direct provider bypass , the emergency client made provider attempt att 103 , but the expected gateway observation was absent. The verdict is BLIND ROUTE , not healthy. The work may be fine; the observability claim is not. In double instrumented attempt , att 105 appears once through the gateway and once through async instrumentation. The verdict is DUPLICATE OBSERVATION . Summing those records would inflate requests, tokens, latency samples, and possibly cost. Deduplicating later by timestamp is weaker than preventing the topology mistake because concurrent calls can look similar. The async failure is different. Helicone's current OpenLLMetry async guide shows provider selection during logger initialization and documents a control that disables all async logging. When that control is off, no traces are sent. The fixture therefore returns LOGGING DISABLED before it tries to infer agent health from an empty query. This precedence keeps evidence honest: A missing record does not prove the call failed. A duplicate record does not prove the call ran twice. Those are coverage findings. Preserve that narrower scope in alerts and incident notes. Keep observation coverage separate from useful completion Once every provider attempt maps to exactly one observation, the dashboard becomes trustworthy for the questions it can answer: which call occurred, how long it took, which model and route were involved, whether the request failed, and how usage changed. Agent health still needs two more ledgers. The work ledger records whether the accepted task is working, waiting, failed, or complete. Activity alone is not progress. A stream of successful LLM calls can repeat the same action with no change in the intended artifact. The outcome ledger checks the promised destination. A report writing agent might require a file with a valid schema and a current run ID. A support agent may require a ticket update at the authoritative API. A deployment assistant may require the expected commit and passing checks. Prefer a deterministic read after write check when the result is inspectable. Consider the fixture's report writer case. It has one async observation for one provider attempt. The application marks the task complete. The expected report receipt is missing. FALSE COMPLETE is the useful verdict because it identifies the exact boundary that failed without claiming the model call was invisible. The approval case is intentionally calmer. The publish step route has a fresh gateway observation, but the work ledger names release manager as the wait owner and supplies a deadline. That is WAITING , not stuck. Page only if the deadline expires, ownership becomes invalid, or the evidence stops refreshing. This three ledger design also prevents one vendor surface from becoming a forced runtime. Helicone can remain the selected LLM observation layer. The application remains responsible for accepted work and destination truth. A separate health layer can correlate those receipts later without becoming a mandatory model gateway. Turn the route audit into a release condition Start with one harmless canary per declared route. Give each canary a unique work id and provider attempt id , send no sensitive content, and write its result to a disposable destination. Then query the observation layer after the documented ingestion allowance. The release condition is: Fail the release on missing or duplicate coverage. Do not silently convert missing evidence to zero traffic. Also fail if a route was removed from the manifest without a matching code or configuration change; otherwise deleting the denominator can make the audit green. Run the same reconciliation continuously with a wider time window: alert on a previously covered route that produces provider attempts with no observation; investigate one attempt appearing through both proxy and async modes; keep evaluation, staging, and production routes distinguishable; expire stale success when the observation query or application receipt is no longer fresh; route legitimate waits to their owner instead of retrying them; verify the destination again after any recovery action. There are limits. The local fixture does not exercise a live Helicone tenant, query permissions, ingestion latency, or retention. A provider attempt ID is application evidence and must be generated and propagated correctly. Exactly one telemetry record is an audit target, not an exactly once execution guarantee. These are reasons to test the contract with canaries, not reasons to trust a nonempty chart. Helicone can provide rich evidence about model requests. The route manifest proves whether that evidence covers the application topology. The work and outcome ledgers decide whether the agent achieved anything useful. Sidewisp is currently in private preview. Its planned territory is agent health across existing runtimes, with evidence, uncertainty, approval boundaries, and outcome verification. Production monitoring adapters and automated recovery are not currently shipped; the private preview waitlist is for teams that want to help shape those checks.