LLM Observability on AWS: Audit AgentCore Span Destinations

The practical answer to LLM observability on AWS is not “open the CloudWatch dashboard.” First prove where Amazon Bedrock AgentCore should deliver spans, then search every destination that can still contain evidence, verify session and trace identity, reject stale observations, and join the trace to a separate receipt for the intended external result. That order matters because AgentCore's destination can change. Current AWS documentation says supported new agents can send spans to a per agent log group, while older configurations may use the shared aws/spans group. ADOT versions before 0.18.0 ignore the unified destination setting. Changing the setting does not move old spans. A query against only today's log group can therefore produce a false “no telemetry” diagnosis even when the missing evidence is exactly where the previous configuration put it. This guide builds a content free audit for that boundary. It uses resource identity, version, destination, timestamps, correlation identifiers, run state, and a boolean outcome receipt. It does not require prompts, responses, tool arguments, or secrets. Locate the evidence before declaring it missing AgentCore Observability provides built in metrics for AgentCore resources and stores metrics, spans, and logs in Amazon CloudWatch. The important boundary is that built in metrics are not the same as application traces. AWS documents default spans for memory resources, while agent runtime and gateway trace detail depends on instrumentation. That creates three separate questions: 1. Is the AWS observation path enabled? CloudWatch Transaction Search must be enabled, and the trace segment destination must be CloudWatch Logs. 2. Where should current spans land? The answer depends on the unified destination setting, Region support, agent age, the execution role, and the ADOT version. 3. Where can historical spans remain? Any destination used before a switch remains part of the investigation window because AWS does not migrate existing span data. The AgentCore configuration guide gives a particularly useful operational boundary: per agent unified delivery requires aws opentelemetry distro =0.18.0 . Earlier versions ignore the configuration and deliver spans to the shared group. The same guide requires permission to install the relevant CloudWatch Logs resource policy. Use those facts to calculate an expected destination before querying: Observation Expected current search scope Operator conclusion Transaction Search disabled None is trustworthy yet Fix setup; do not infer agent health Trace segments not routed to CloudWatch Logs None is trustworthy yet Fix the destination prerequisite Unified requested, ADOT below 0.18.0 Shared aws/spans A blank per agent group is a query scope error Unified active and role policy allowed Per agent runtime log group Check current spans there Destination changed during the review window Current and previous groups Search both; old spans stay where they landed This table is deliberately not a single “telemetry present” check. A missing record in the per agent group can mean setup failure, blocked delivery, an old ADOT version, or a correct historical record in the shared group. Those states need different repairs. Keep a destination transition record Do not make the active setting your only source of truth. Store a small transition record beside the runbook: The record contains no prompt or response content. It answers the query planning question that a dashboard cannot reconstruct later: which destinations overlap the incident window? Run a migration aware AgentCore audit The audit used for this article evaluates eleven fixed cases. Its input contract is intentionally small: Its decision order is more important than its syntax: Running the classifier over the fixture produced: The cases cover disabled Transaction Search, the wrong trace destination, old ADOT queried only in the per agent group, omitted historical evidence after a switch, insufficient delivery authority, a missing current span, stale evidence, broken correlation, a legitimate approval wait, false completion, and a healthy migration aware result. This is a decision test, not proof about a live AWS account. Adapt its inputs from your own configuration and canary queries. Preserve the ordering: otherwise a generic telemetry missing verdict can hide the much more actionable fact that the operator searched the wrong place. Preserve session identity, trace identity, and freshness AWS describes AgentCore observability as a hierarchy: a session contains traces, and a trace contains spans. The telemetry documentation makes that hierarchy explicit. It is useful only if identity survives the request path. For ADOT instrumented AgentCore runtime calls, the configuration guide documents two propagation details: send X Amzn Bedrock AgentCore Runtime Session Id so the session ID reaches downstream telemetry; invoke the runtime with traceId=<traceId when a trace ID must be propagated. Record whether those identifiers are present, not their sensitive payload context. A span with no joinable session can still prove that code executed, but it cannot support a session level incident timeline. Classify that as correlation broken , not healthy. Freshness needs a similarly explicit contract. A trace found last week does not prove that delivery works now. Define: Choose the maximum age from the workflow's expected cadence and incident tolerance. Five minutes is reasonable for an every minute canary; it is unreasonable for a nightly batch. Store the threshold with the verdict so “fresh” remains inspectable. Waiting also needs evidence. If a trace shows a bounded approval dependency with an owner and the run is resumable, return waiting . Do not page it as stuck merely because no new tool span appeared. If the approval record is absent, contradictory, or expired, return uncertain or escalate according to the runbook. Require an outcome receipt after the trace A complete trace answers “did the instrumented execution path finish?” It does not necessarily answer “did the intended work happen?” The distinction is visible in common failures: an upload tool returns before the destination commits the object; a message API accepts a request but the message never reaches the intended channel; an agent writes a local file while the required artifact belongs in remote storage; the final model call succeeds after a downstream transaction has already rolled back; an approval wait is incorrectly converted into terminal success. AWS prescriptive guidance recommends correlating LLM evidence with downstream impact. A privacy minimal implementation can do that with an outcome receipt: The receipt should be produced by the strongest deterministic check available: an object HEAD , a database read by stable key, a public API fetch, a checksum, or a targeted test. It should not contain the object body, prompt, response, or secret. Keep the two verdicts separate: Trace evidence Outcome receipt State Missing or stale Any Observability evidence is insufficient Complete Missing false complete Waiting on recorded approval Not expected yet waiting Complete and fresh Present and verified healthy for this tested outcome The final row is scoped. It proves the fixed canary and destination, not every route, every task, or semantic output quality. Adopt the audit without collecting content A useful production receipt needs only enough data to distinguish the failure layers: agent resource and endpoint identifiers in a redacted or hashed form; Region and observation time; Transaction Search and trace destination status; ADOT version and unified destination setting; current and previous destination classes; whether both destinations were searched for the investigation window; newest matching canary time; presence of session ID and trace ID; run state and bounded approval state; stable operation identity and deterministic outcome receipt status. Keep prompt text, model responses, tool arguments, credentials, raw headers, and customer payloads out of this receipt. If deeper content inspection is required for a specific incident, authorize and scope it separately. The audit also has limits. It does not prove instrumentation coverage across every application route, sampling completeness, CloudWatch retention, export recovery, or semantic answer quality. It proves that the selected evidence path is configured and searchable, the canary is fresh and correlated, and the selected external outcome has its own receipt. That is enough to prevent an expensive category error: changing the agent because an operator searched the wrong span destination. Sidewisp is currently in private preview. Its intended role is to turn evidence like destination coverage, freshness, correlation, waiting state, and outcome verification into a clear health view. Production AgentCore and CloudWatch monitoring adapters are not currently shipped, so this article is an operating pattern you can apply now—not a claim that Sidewisp already runs this audit.