LLM Observability Without Prompt Leakage: A Safe Telemetry Contract

LLM observability should explain which operation ran, what failed, how long it took, whether it retried, and whether the intended result exists . It does not need to copy every prompt, tool argument, or tool result into a telemetry backend to answer those questions. A safe production default is a content minimized envelope: correlation IDs, operation and tool names, low cardinality error classes, timing, attempt number, input key names , result shape, and an independent outcome verdict. Capture message or tool content only for a specific investigation, with explicit authority, a short retention window, and storage controls that match the data. This guide builds that default and tests it against three deliberately sensitive agent events. Trace the operation, not the private payload The broad definition of LLM observability includes traces, metrics, logs, evaluations, model behavior, and application outcomes. For an agent operator, the useful question is narrower: can the evidence distinguish a slow model call, a denied tool call, a legitimate wait, a retry loop, and a tool that returned successfully while the deliverable never appeared? Current OpenTelemetry GenAI semantic conventions offer a useful starting vocabulary. The execute tool span uses gen ai.operation.name and gen ai.tool.name ; it also describes a tool call ID, tool type, and error.type . The same document marks tool call arguments and results as opt in and warns that both may contain sensitive information. These conventions are in Development , so pin the version you implement and expect changes. That split is operationally sound. The following fields usually answer the first incident response questions without reproducing content: Field What it answers Collection rule trace id , run id , step id Which execution and step does this event belong to? Keep opaque identifiers; do not derive them from content. operation , tool name , tool type What kind of work was attempted? Use bounded, documented names. status , error type Did it fail, and by what stable class? Prefer timeout or permission denied over a free form provider message. duration ms , attempt Is it slow or repeating? Record numbers, not a narrative. input keys What shape of request was attempted? Keep key names; discard values by default. result shape Did the tool return nothing, an array, or an object? Keep type, count, or top level keys; discard values. outcome check , outcome status Did the user's expected result land? Verify separately from tool completion. content captured Is content present elsewhere? Make the boundary visible and queryable. run id and the outcome fields above are an application contract, not OpenTelemetry standard attributes. That is deliberate: tracing vocabulary and task specific health evidence solve related but different problems. An allowlist beats an endless redaction list Redaction code asks, “Which dangerous values have we remembered to remove?” An allowlist asks, “Which fields have earned the right to leave the process?” For routine production telemetry, the second question has a smaller failure surface. The OWASP Logging Cheat Sheet says access tokens, passwords, database connection strings, encryption keys, and sensitive personal data usually should not be recorded directly. It also calls out file paths and internal network details as data that may need special treatment. Agent tools routinely handle every item on that list. Free form error messages are a common back door. This is unsafe: The operationally useful version is smaller: It still supports aggregation, alert routing, retry analysis, and correlation to a protected local record. It does not turn the observability store into a copy of the secret bearing error. Key names require judgment too. calendar id reveals less than its value, but a key such as customer alice hiv status already contains content. Keep an approved schema for tool names and key names; reject or map unexpected names to other . Cardinality is not only a cost concern—it is a data boundary concern. Reproduce the sanitizer on three hostile fixtures The accompanying fixture contains three fake but realistic events: 1. a successful calendar lookup with a fake email and bearer token; 2. a denied preview publication with a fake API key and private path; 3. a timed out database write with a fake connection string and email. The sanitizer does not recursively hunt for secret patterns. It constructs a new object from an allowlist: Run the checked artifact from the article's evidence bundle: The reproduced output retains the diagnosis: It also supports concrete conclusions. The calendar tool completed and its outcome check passed. The preview call needs permission attention, not a blind retry. The waitlist write timed out on its second attempt and the durable record check failed, so “the command returned” is not a success condition. The byte for byte diff passed. A separate search for all planted canaries—bearer token, API key, connection string, email values, and private path—returned no matches in the sanitized file. That is a useful test to keep in CI: add a unique fake canary to every sensitive fixture and fail if it crosses the exporter boundary. Content capture is an escalation mode Content free telemetry has a real limitation: it cannot explain why a prompt was semantically misunderstood or whether a model response contained a subtle factual error. The boundary should therefore be a controlled escalation, not an absolute claim that content is never useful. OpenTelemetry's content capture guidance says instructions, user messages, and model outputs are sensitive and often large, and should not be captured by default. It describes separate external storage with distinct access controls as a production option when sensitive content must be retained. Before enabling that mode, decide all of the following: the exact run, tool, or tenant in scope; who approved collection and who may read it; which fields are filtered before export; the maximum duration and retention; the storage region and access controls; how deletion is verified; how the mode automatically returns to content captured: false . Sampling is not redaction. A one percent sample can still capture the one prompt containing a credential. Encryption is not minimization either: it protects stored bytes but does not justify collecting them or prevent authorized overexposure. Start with the allowlist, then add a separately governed forensic path only when the operational question cannot be answered otherwise. Join telemetry to an outcome, then decide An observability event describes an operation. Agent health requires one more fact: did the intended work become true? A database driver can return success while a later transaction rolls back. A file writing tool can exit zero while writing the wrong path. A message API can accept a request while the destination rejects it asynchronously. The telemetry envelope should therefore carry the name and status of a deterministic outcome check without copying the outcome's content. Use this decision order: 1. Outcome passed: close the operation even if a transient retry occurred; retain the retry signal for cost and reliability analysis. 2. Tool failed and outcome failed: route by the stable error class. Retry only if the operation is reversible and within an approved limit. 3. Tool succeeded and outcome failed: classify false success and investigate the contract between the tool and verifier. 4. Outcome not run: do not call the work complete. Schedule or perform the check. 5. Evidence is missing or contradictory: mark the state uncertain and ask before an active intervention. This separates activity from useful progress and command completion from the user's result. It also makes recovery safer: an operator can prepare one bounded action, then clear the issue only after the outcome check passes. The production default For most LLM and agent workloads, begin with a small, documented telemetry allowlist. Keep correlation, operation, stable error, timing, attempt, shape, and outcome fields. Keep content off by default. Test the exporter with planted canaries. Treat temporary content capture as a separately authorized incident procedure. That default will not answer every forensic question. It will answer routine health questions without silently creating a shadow database of prompts, credentials, customer records, and tool results. Sidewisp's product direction is to turn availability, execution, memory, tool, outcome, and cost evidence into a clear health view while preserving human authority and collecting only what is needed. Sidewisp is currently in private preview. The public site and articles are live; production monitoring adapters and automated recovery are not generally shipped. If this evidence boundary matches how you want to operate agents, you can join the private preview waitlist.