Honeycomb LLM Observability: Audit the Agent Timeline Contract

Honeycomb can show a polished Agent Timeline and still be working from a broken event contract. Before using that view to diagnose a multi agent incident, verify six things in the emitted spans: one conversation identity, distinct agent names, caller side invocation attribution, recognized operation names, stable tool call IDs, and propagated errors. The practical test is not “did telemetry arrive?” It is “could these fields project the work into the right conversation, agent lane, operation, and failure state?” The audit below answers that narrower question with eight content free fixtures. It does not collect prompts or claim that a clean timeline proves the requested external result. What Honeycomb Agent Timeline actually projects Honeycomb documents Agent Timeline as a view over entire conversations that can span multiple traces. The conversation key is gen ai.conversation.id . Inside a conversation, GenAI spans are grouped by gen ai.agent.name , then presented as agent invocations, LLM operations, and tool calls according to gen ai.operation.name . That makes the view useful, but it also makes several application supplied fields part of the diagnostic boundary: gen ai.conversation.id decides which spans belong together. gen ai.agent.name decides which agent owns a lane. gen ai.operation.name decides whether a span is treated as chat, agent invocation, or tool execution. gen ai.tool.call.id lets an operator correlate a tool request with its result. error status and exception fields decide whether a failure is visible at the affected span and its parent. Honeycomb's instrumentation guide marks conversation ID, agent name, and operation name as required for its agent view. It also says a missing agent name appears as Unknown , and that duplicate names prevent operators from distinguishing agents during an investigation. The same guide makes an easy to miss attribution rule explicit: the calling agent emits the invoke agent span. The called agent emits its own chat , execute tool , and other spans under its own unique name. If a collector transform assigns the invocation to the callee, the timeline may contain every span yet tell the wrong causal story. This is a schema conformance problem before it is a dashboard problem. Querying harder cannot recover an identity that was never emitted or correct an agent name that two workers share. Audit the projection contract before an incident The companion fixture contains eight small span sets. None contains prompts, responses, tool arguments, customer identifiers, or secrets. The only values are synthetic conversation IDs, agent instances, agent names, operation names, parent links, tool call IDs, and status fields. Run the audit from the article artifact directory: The classifier applies the checks in causal order: 1. All spans in the proposed conversation must share one nonempty conversation ID. 2. Every span must have a nonempty agent name. 3. One displayed agent name must not belong to multiple agent instances. 4. An invoke agent span must be owned by the caller. 5. The operation must belong to the documented GenAI operation set used by the projection. 6. An execute tool span must retain a tool call identifier. 7. A child error must not sit below a parent that still reports success. The order matters. If one conversation is already split across conv 201 and conv 202 , counting agent lanes inside either fragment gives a precise answer to the wrong question. Likewise, if two processes both call themselves worker , tool call correlation cannot restore the missing agent identity. The rules are intentionally stricter than “Honeycomb rendered something.” A rendered view is an observation. A conformant span envelope is evidence that the observation was grouped and attributed according to a declared contract. Eight fixtures expose seven different false greens The naive comparison rule only checks that every span has a conversation ID. It accepts all eight fixtures—including the case where two different nonempty IDs split one logical conversation. The conformance audit accepts exactly one: Fixture Naive ID check Conformance verdict Required repair valid conversation pass CONFORMANT none fragmented conversation pass FRAGMENTED CONVERSATION propagate one conversation ID across trace boundaries unknown agent pass UNKNOWN AGENT emit a stable, nonempty agent name agent name collision pass AGENT NAME COLLISION give each agent instance a distinct operational name wrong invoke attribution pass INVOKE ATTRIBUTION INVALID emit invoke agent from the caller unrecognized operation pass OPERATION UNRECOGNIZED remap to a documented GenAI operation missing tool call id pass TOOL CALL UNCORRELATED preserve the call ID across request and result child error hidden pass ERROR PROPAGATION BROKEN propagate failure status to the parent Each verdict protects a different operator decision. A fragmented conversation hides work. An unknown or colliding name corrupts ownership. Wrong invocation attribution reverses who delegated to whom. An unrecognized operation drops an event into the wrong visual category. A missing call ID makes a timeout unsafe to reconcile. A hidden child error turns a failed tool path into a successful looking parent. The last case deserves special attention. Honeycomb advises recording errors consistently and propagating a tool call failure to the parent span. That propagation does not prove whether an external side effect committed; it does prevent the parent from silently remaining green while a child is known to have failed. For effectful tools, add a separate destination receipt after this telemetry check. A timeout plus no response is an uncertain effect, not permission to retry. The Agent Timeline can help locate the ambiguous call; the authoritative destination must decide whether the effect exists. Turn the audit into a deployment canary Run the fixture locally first, then adapt one harmless canary conversation to your instrumentation path. Keep the canary content free and make its topology obvious: one router agent invokes one worker; the worker performs one harmless tool call; every span receives the same synthetic conversation ID; router and worker have distinct names; the invocation belongs to the router; the tool call retains one synthetic call ID; one test variant fails the tool deliberately and checks that the parent is no longer successful. Compare the emitted envelope before export and after any OpenTelemetry Collector transform. This catches a common boundary failure: source instrumentation is correct, but a transform renames spans, drops an attribute, or assigns one static agent name to every process. Pin that transform. The OpenTelemetry GenAI agent semantic conventions inspected for this article are marked Development , so an SDK or Collector upgrade is not a routine visual change. It is a schema change that should rerun the canary. Use a small acceptance record for each release: Do not put prompts, responses, tool arguments, or tool results into this record. Honeycomb notes that message content can contain PII and recommends putting such content in span events where a Collector can filter it. The conformance canary needs identities and status, not conversation content. Where this test stops Passing the audit means the supplied span envelope can project coherently into Honeycomb Agent Timeline. It does not establish that: every span reached Honeycomb; sampling retained the critical failure; an agent is making useful progress; a legitimate human wait has an owner and deadline; a tool side effect committed exactly once; the promised file, ticket, deployment, or report exists; the answer is factually or semantically correct. Those are separate health questions. Treat timeline conformance as the entrance ticket for diagnosis, not the final health verdict. After it passes, add freshness, waiting state, effect, and outcome receipts according to the workflow's risk. This boundary is also why the article's experiment does not compare Honeycomb with another vendor. The reader decision is earlier and more concrete: can I trust the conversation and agent attribution I am about to investigate? Sidewisp is currently in private preview. It is intended to add a health layer around existing agent runtimes, but production agent health collection, Honeycomb adapters, and automated recovery are not shipped in the current website repository. If you are defining the evidence boundary for your agents now, the private preview waitlist is the appropriate way to share that integration need. Sources Honeycomb: Instrumenting AI Agents Honeycomb: Agent Timeline OpenTelemetry GenAI agent semantic conventions at inspected commit