LangSmith LLM Observability: Prove the Trace Tree Across Services

LangSmith can show a clean set of successful spans and still leave an operator with the wrong conclusion. The missing evidence may be a worker whose span never joined the trace, a second root created after a service hop, or a completed trace with no proof that the requested artifact reached its destination. The practical default is simple: treat one accepted agent request as one trusted trace tree , then test that tree against an expected service manifest. A healthy verdict requires one root, resolvable parent links, the expected service stages, fresh progress or an explicit wait, and a separate destination receipt. Trace completion alone is not task completion. This guide turns that rule into an eight case audit for langsmith llm observability . The fixture contains no prompts, responses, secrets, or tool payloads. It compares a naive “every span says ok” rule with a stricter operational decision and exposes five false green cases. The default: preserve one root across trusted service hops LangSmith defines traces as trees of runs, and its distributed tracing documentation describes how a request can continue across services. The originating service serializes the current run tree into a langsmith trace header, with optional baggage for metadata and tags. The receiving internal service restores that context and continues the trace. That is the reasonable implementation default for an API that hands work to a planner and then to a worker. It gives the operator one causal tree instead of unrelated local traces. The same documentation puts an important boundary beside the feature: those headers are trusted tracing context. A public endpoint must not accept them as an authoritative parent or accept arbitrary baggage from an untrusted caller. Strip them at the edge, start a new root for the accepted request, and propagate the newly created context only across trusted service to service hops. For each accepted request, record a content free manifest such as: requestKey should be an opaque identifier, not a prompt or customer name. The manifest is the denominator: without it, a trace cannot tell you that an expected service emitted nothing. Audit structure before evaluating span status Start with structure because a collection of locally successful spans can still describe the wrong execution. Apply these checks in order: Check Evidence Failure state Trust boundary The root was created after the public edge; propagated baggage came only from an internal service UNTRUSTED CONTEXT Root count Exactly one root maps to the accepted request SPLIT TRACE Parent integrity Every non root parent run id resolves inside the same trace ORPHAN SPAN Service coverage Every required manifest stage has at least one expected run TRACE GAP Freshness Open work has recent progress or a valid dependency deadline STUCK or WAITING Outcome The named destination has a deterministic receipt OUTCOME UNVERIFIED Final verdict All applicable checks pass HEALTHY This ordering matters. If a public caller can supply trusted baggage, the tree is not safe evidence even when its shape is perfect. If two roots exist, checking children under only one root can hide half the execution. If a parent ID is absent, timestamps do not repair the causal gap. Do not force a running trace to include stages it has not reached. Service coverage is evaluated at the relevant lifecycle boundary. Before the planner dispatches work, absence of a worker is normal. After the run reports completion, the same absence is a trace gap. Store the lifecycle milestone beside the manifest instead of inferring it from wall clock time alone. Retries also need an explicit rule. A second worker span is not automatically a duplicate failure. Record a stable operation key and attempt number, then ask whether the retry remained under the same accepted root and whether only one effect was committed. This trace tree audit checks causal coverage; an idempotency or effect receipt audit handles duplicate side effects. Replay eight inconvenient traces The research artifact uses four required services— api , planner , tool worker , and verifier —and replays eight cases: 1. a complete distributed trace with a verified artifact; 2. a missing worker span; 3. a worker whose parent ID is absent; 4. two roots for one accepted request; 5. a fresh wait with an owner and future deadline; 6. an open worker with stale progress; 7. a complete trace with no destination receipt; 8. baggage accepted from a public propagation source. Run the deterministic classifier with Node: The exact fixture produced: The falsifiable result is the gap between six naive greens and one verified healthy run. Five cases contained only ok spans yet failed a structural, trust, or outcome check. The stale worker was not naive green because it remained open. The legitimate wait also remained open, but it was routed to WAITING rather than mislabeled as failure. This does not mean seven agents were broken. It means only one case had enough evidence for a healthy conclusion. WAITING is a valid operational state, and the gap states identify what evidence must be restored before deciding. A compact classifier can implement the precedence: The precedence preserves the difference between activity and useful progress. It also prevents a later green condition from erasing an earlier evidence failure. Query the tree without exporting prompt content LangSmith’s official SDK query guide recommends list runs or the /runs/query API. It documents is root=True for root run selection and read run(..., load child runs=True) for hydrating a candidate tree when a local child predicate needs fields that server side filters do not expose. A minimal Python collection path is: Adapt the select list to the installed SDK. The research snapshot observed the official langsmith sdk release v0.10.11, published on July 28, 2026; verify the current reference before copying field names into production code. The audit does not need inputs or outputs . Traverse the hydrated child tree and retain only: run and trace IDs; parent run ID; an application emitted service or stage tag; start, end, and last progress times; status or error presence; opaque operation and request keys; wait owner and deadline; the destination receipt’s identifier and verification result. The LangSmith observability overview correctly separates trace investigation, dashboards, alerts, automations, online evaluations, and feedback. A structural trace audit complements those surfaces; it does not replace evaluations or claim that every failure is visible from metadata. Keep waiting, stuck, and outcome verification separate A long running span is not enough to call an agent stuck. Require both a freshness breach and the absence of a valid wait contract. A wait contract needs an owner, a dependency, and a deadline. In the fixture, the worker had recent progress, waited on a reviewer, and carried a future deadline. The correct state was WAITING . The stale worker had no such explanation and had exceeded the five minute freshness window, so it became STUCK . Neither state is HEALTHY . Waiting needs routing to the named owner; stuck work needs investigation and perhaps one bounded recovery proposal. Do not retry merely because a trace stopped moving, especially near a side effect boundary. Finally, require evidence outside the trace. A verifier span that ended successfully proves that code ran. It does not prove that the requested file exists, a deployment reached the intended revision, or a message reached the intended destination. Define the strongest deterministic receipt available: an immutable object hash, a database row plus invariant, a remote revision, or an API read back. The limitation is deliberate. This audit cannot reconstruct spans that were never emitted, judge semantic quality from metadata, or prove a tool effect without an application produced receipt. It can show that the evidence is incomplete and keep uncertainty from becoming green. Sidewisp uses this same health first boundary in its product direction: traces are evidence, while useful progress and expected outcomes require their own checks. Sidewisp is currently in private preview. Its live site is an early access experience and interactive demonstration; production agent health collection and automated recovery are not generally shipped. Join the private preview if you want to help shape a health layer that works alongside existing runtimes rather than replacing LangSmith or your agent stack.