Azure LLM Observability: Audit the Sampled Green Gap

Azure LLM observability should answer more than “did a run finish?” Before accepting a healthy verdict, prove four conditions for the same run: the telemetry is queryable and fresh, the execution state is understood, every required quality check actually covered the run, and the promised result exists at its destination. Microsoft Foundry gives you useful pieces of that evidence. It can place server side traces in Azure Monitor Application Insights, show operational metrics in the Agent Monitoring Dashboard, and run evaluations against sampled production responses. Those pieces are not interchangeable. A trace can be fresh while an evaluator skipped the run. An evaluator can pass while a file, ticket, deployment, or message never reached its destination. A completed run can also be waiting on a legitimate human decision that the application modeled poorly. The practical default is therefore a coverage audit, not a composite score. Keep unavailable evidence unavailable, treat sampling as coverage rather than success, and let a workload specific outcome receipt close the final gap. Read the Foundry surfaces as separate evidence Microsoft’s observability overview separates three capabilities: Tracing records the execution path, including model calls, tool use, latency, and related spans. Monitoring summarizes operational measures such as tokens, latency, error rate, and run success. Evaluation measures selected quality or safety properties with built in or custom evaluators. That separation matters during an incident. A successful span establishes that an instrumented operation reached a terminal status. It does not establish that the current dashboard can see all relevant spans, that an evaluator examined this response, or that the requested side effect happened. Foundry’s tracing setup guide makes two useful boundaries explicit. First, hosted agent server side tracing begins after the project is connected to Application Insights. Second, new traces can take a few minutes to appear. If the expected trace is absent, the responsible verdict is not “failed” or “healthy.” It is evidence unavailable until you distinguish connection, authorization, ingestion delay, sampling, and instrumentation. Query access is another independent condition. Foundry’s monitoring documentation requires suitable Azure role based access to Application Insights and, for log views, the associated Log Analytics workspace. An operator who can open the project but cannot query its protected telemetry has a visibility problem, not evidence of a healthy agent. The same caution applies to the Monitor tab. The Agent Monitoring Dashboard guide describes run success rate, tokens, latency, and evaluation outcomes. It also says continuous evaluation runs on sampled responses. Sampling is a valid cost and throughput decision, but it creates a denominator question: did this particular run receive every evaluation required for the decision you are making? Do not answer that question from an aggregate score. Record it per run. Give one run a coverage record Start with a small, content free record. Keep identifiers or hashes that let an authorized operator find the underlying evidence; do not copy prompts, tool arguments, secrets, or model outputs into a new health store. Each field answers one decision: 1. Can the operator retrieve current telemetry? Test the Application Insights connection and actual query permission. A portal page loading is not the test. 2. Is the trace fresh enough for this workflow? Set a budget from the run’s expected duration plus observed ingestion delay. Do not silently reuse yesterday’s green span. 3. What is the agent doing? Preserve working , waiting , succeeded , and failure states. A named approval or external dependency is a wait, not a stall. 4. Did the required evaluator cover this run? Store coverage separately from the evaluator’s result. 5. Did the promised outcome occur? Query the destination that owns the result. The outcome check must match the work. For a generated report, verify the expected object exists and its hash or schema is correct. For a ticket update, read the ticket and check the intended state transition. For an API mutation, query the target resource rather than trusting the client’s successful HTTP exchange. For a code task, require the expected diff plus the relevant build or test result. This record deliberately avoids a universal “success” field. Combining unlike evidence too early is how unknown coverage becomes green. Replay the decision before wiring alerts The fixture used for this article contains eight synthetic runs and no Azure credentials, prompts, or production telemetry. Its classifier evaluates visibility before run state, run state before quality, and quality before outcome: Running node audit azure observability.mjs against the fixed fixture produced eight matches and no mismatches: Case Azure side evidence Destination evidence Verdict Connection present, query denied Cannot inspect current telemetry Receipt present EVIDENCE UNAVAILABLE Old trace, all other fields green Stale Receipt present EVIDENCE STALE Fresh trace, run active Current activity Not expected yet WORKING Fresh trace, named approval wait Current wait Not expected yet WAITING Run succeeded, required eval skipped Quality coverage absent Receipt present QUALITY UNKNOWN Run succeeded, sampled eval failed Quality failed Receipt present QUALITY FAILED Run and eval succeeded Complete in Foundry Receipt absent FALSE COMPLETE Run and eval succeeded Complete in Foundry Receipt present HEALTHY Two results are easy to mishandle. QUALITY UNKNOWN is not a failed evaluator. It says the evaluator did not cover the run required for this decision. You can route that state to a deterministic substitute, a one off evaluation where appropriate, or a human review. You cannot relabel the aggregate dashboard score as this run’s result. WAITING is also not a failure. If the trace is fresh and identifies a legitimate owner and dependency, the useful action is to surface the wait to that owner. Restarting the agent can duplicate work or discard context without resolving the dependency. Set alerts on the failed condition, not the color An alert should name the evidence that broke: Telemetry unavailable : verify the Foundry to Application Insights connection, query RBAC, protected table access, instrumentation, and recent traffic. Evidence stale : compare the last observed trace time with the workflow’s freshness budget and known ingestion delay. Quality unknown : inspect the configured sample rate and whether this decision actually requires an evaluator. Quality failed : preserve the evaluator name, version, threshold, and the tested run identifier before investigating. False complete : stop automated retries at the side effect boundary and reconcile the destination by a stable work identifier. This produces quieter operations than an alert on every missing sample or long duration. Microsoft’s dashboard guidance offers broad investigation thresholds, such as looking into low run success rates or high latency. Those fleet signals are useful for finding a cohort. The per run coverage record decides what is wrong with a particular piece of work. Freshness also needs an owner. Application Insights retention controls how long evidence remains queryable; ingestion and query permissions control whether it is visible now. Store the last successful query time and the newest matching trace time separately. “The dashboard loaded” proves neither. Keep the preview and privacy boundaries visible Foundry’s current documentation marks parts of agent tracing and monitoring as preview. The agent tracing overview says tracing is generally available for prompt and hosted agents while workflow and external agent tracing are in preview. The monitoring guide also marks dashboard features as preview. Record the agent type and feature status in the runbook; do not transfer guarantees from a hosted agent path to an external workflow without checking the current contract. Tracing can capture prompts, outputs, tool arguments, and tool results. Microsoft advises redacting sensitive content before it reaches telemetry and applying production access and retention controls. A health layer should reference the evidence with content free identifiers where possible, not create a second store of sensitive payloads. There is one final limitation to keep beside the verdict: this classifier tests evidence precedence. It does not contact an Azure subscription, infer an ingestion service level objective, or decide what success means for your application. The destination receipt is deliberately workload specific. That boundary is the point. Azure LLM observability can expose execution, performance, sampled quality, and debugging evidence. Operational health also requires freshness, coverage, a correct waiting state, and proof of the expected result. Sidewisp is currently in private preview. Its direction is to turn evidence from existing agent runtimes into a clear health view while preserving uncertainty and human approval boundaries; Microsoft Foundry monitoring is not presented here as a shipped Sidewisp integration.