Phoenix LLM Observability: Prove Traces Survive Restart

Phoenix can show a complete trace while its own evidence layer is still fragile. A reachable page proves that the web process answers now. It does not prove that an older trace survived a restart, that the collector resumed ingestion, or that the effective retention policy covers the period in which your team investigates incidents. The reasonable default is a two canary restart drill: 1. query one content free trace created before the restart; 2. restart the Phoenix service without changing application code; 3. query that same trace again; 4. emit and query a second trace after the restart; 5. compare the observation age and effective retention with explicit limits. The old canary tests persistence. The new canary tests resumed ingestion. You need both. If only the old trace exists, storage may be fine while collection is broken. If only the new trace exists, the server has come back on empty or unexpected storage. Treat Phoenix as three evidence layers Phoenix's architecture documentation separates the system into a web interface, a trace collector, and a SQL database backend. That distinction matters during diagnosis: the interface can answer while OTLP ingestion fails; the collector can accept a connection while a trace never becomes queryable; the database can be reachable while the container points at a new SQLite working directory; all three can be up while a retention job removes evidence sooner than the incident process expects. Phoenix supports SQLite and PostgreSQL. Its current documentation positions SQLite for local development and single user deployments, with data under ~/.phoenix/ or PHOENIX WORKING DIR . PostgreSQL is the documented production choice for multi user and high availability deployments. This is not a rule that SQLite is always unhealthy. A single developer can run a reliable local instance with a mounted volume. The failure is leaving persistence implicit. The official Docker guide shows the two contracts directly: For PostgreSQL, Phoenix reads PHOENIX SQL DATABASE URL ; the guide documents PostgreSQL 14 or newer. Keep the connection value in your secret system, not in a health receipt. The receipt needs only the backend class, an opaque deployment identifier, and the result of the canary queries. Pinning the image is a separate control. latest may be convenient for a disposable local trial, but it makes a restart capable of changing the application and its database expectations at the same time. Record an immutable image digest or explicit version before the drill. A successful restart against an unknown image is not reproducible evidence. Build one content free restart receipt Choose a canary path that exercises the same collector and project routing as the agent traffic you care about. Do not put a real prompt, model response, tool argument, credential, or customer identifier in the canary. A random run label and timestamps are enough. Phoenix's documented REST endpoint can list traces for a project with start time bounds and optional spans. Use your deployment's authentication method, but keep the authorization value out of shell history and saved output. For example, set non secret routing values and request a narrow time window: Search the response locally for the canary's opaque trace id ; do not export the response body as general telemetry. If you request include spans=true , response size and query latency increase, so the Phoenix API reference recommends fetching span detail lazily. The restart drill needs trace identity and time, not prompt content. A minimal receipt can look like this: effectiveRetentionDays means the policy attached to the actual project, not merely a deployment default. Phoenix retains data indefinitely by default, represented as zero days in its default policy. An administrator can assign time or trace count based policies to individual projects. The deployment time default can update new projects without changing existing project specific overrides. That is why configuration intent and effective project state are different evidence. Compare retention with your operating process. If an incident can sit unnoticed for 14 days, a seven day trace window is degraded even when every current trace is present. Thirty days is not intrinsically healthy either; it is healthy only relative to the review window, storage budget, and data governance decision. Run the drill without hiding the interruption Use the runtime's normal restart operation. Do not combine the first drill with a Phoenix upgrade, storage migration, collector reconfiguration, or application release. The point is to isolate persistence and resumed ingestion. Immediately before restart: 1. record the pinned version or image digest; 2. confirm the intended database backend and durable volume or database identity; 3. record the effective project retention policy; 4. emit the first canary through the normal instrumented path; 5. query it and store only the boolean result, trace ID, and timestamps. After restart: 1. wait for the service's documented readiness behavior rather than using an arbitrary sleep; 2. query the same pre restart trace; 3. emit a different post restart canary; 4. query the new trace through the same project route; 5. stamp the receipt and evaluate it before its freshness limit expires. The companion fixture for this article applies a five minute observation limit and replays nine states: The result is 9/9 cases pass . Two configurations classify as healthy: pinned PostgreSQL for a multi user deployment, and pinned SQLite with a durable volume for one user. The remaining fixtures deliberately produce evidence lost , ingestion failed , waiting , needs human , uncertain , or degraded . The decision precedence matters: Evidence Verdict Operator decision Migration is running as planned waiting Observe the bounded maintenance operation; do not call it an agent failure Migration failed needs human Stop automatic recovery and inspect the database/version boundary Observation is older than the freshness limit uncertain Re run the query before acting In policy old trace disappeared evidence lost Preserve current storage and diagnose mount/database identity first Old trace remains but new trace is absent ingestion failed Inspect collector reachability, exporter path, auth, and project routing Both traces exist but retention is too short degraded Align the effective project policy with incident review Both traces exist, evidence is fresh, and controls match healthy The evidence layer passed this bounded drill This ordering prevents a configuration warning from masking actual data loss. An unpinned image is important, but a missing in policy trace is the first incident. Put migrations outside blind recovery Phoenix documents that new major versions may run database migrations during startup. It also warns that rolling an application image back does not automatically downgrade the database schema. That makes “restart the old container” an unsafe generic recovery rule after a failed major upgrade. For Kubernetes, the migration guide recommends running migrations in an initContainer so they complete before the main container is subjected to liveness checks. It also explains that PostgreSQL index creation can block writes. PHOENIX MIGRATE INDEX CONCURRENTLY=true avoids holding that write lock, but the documented trade off is a migration roughly two to three times slower, and the new pod still waits for completion. Translate those mechanics into states: a migration running inside its approved maintenance window is waiting ; a query taken while migration state is unknown is uncertain ; a failed migration that may have advanced the schema is needs human ; automatic rollback is allowed only when the database compatibility plan explicitly supports it. This is the same distinction reliable agent operations need elsewhere: activity is not progress, waiting is not stuck, and command completion is not the intended outcome. Know what this receipt does not prove A passing restart receipt protects one evidence path. It does not prove that every model route is instrumented, every span is semantically correct, a backup can be restored, or an agent delivered the intended result. It also does not validate the content of an LLM evaluation. Those are separate tests. The receipt is intentionally content free. That reduces exposure, but it also means semantic errors require a bounded evaluation or human review. Treat unavailable evidence as unavailable; do not fill it with a healthy guess. Sidewisp's intended health model is concerned with evidence freshness, reachability, useful progress, outcomes, and safe recovery boundaries. The operational pattern here fits that territory, but it is not a claim of a shipped integration. Sidewisp does not currently connect to Phoenix or monitor this deployment. Sidewisp is currently in private preview. If you are defining the first health contract for an agent stack, keep this restart receipt beside the agent's outcome receipt. One tells you whether the diagnostic evidence survived. The other tells you whether the work did.