Opik LLM Observability: Audit Thread Scores Before Green

Opik can tell you a great deal about a multi turn agent, but a visible trace or a high conversation score is not yet a health verdict. The reasonable default is to use Opik for trace and evaluation evidence, then require four additional facts before showing green: the intended turns landed under one thread identity, the thread was eligible for scoring, the score was produced after the latest activity, and the requested result exists at its destination. That distinction matters most when a score is missing or looks reassuring. “No score” may mean the conversation is still active, the sampling rule excluded it, scoring is pending, or scoring has stalled. A score of 0.94 may belong to the previous version of a thread. Even a fresh 0.94 can coexist with a missing file, unsent message, or failed update. This guide builds a content free receipt and replays ten cases against it. It was checked against Opik 2.2.12 at repository commit c54a6a9 on July 29, 2026. It does not require prompts, responses, credentials, or customer identifiers. Prove the thread before judging the score Opik groups related traces with a user defined thread id . Its pinned conversation documentation says that the identifier must be unique within a project. That gives operators an important boundary: a conversation is not “whatever rows happen to look related” in the dashboard. Before reading any thread level evaluator result, record: the workspace and project expected to receive the traces; an opaque hash or non sensitive representation of the expected thread ID; the distinct thread IDs observed for the intended turns; the latest trace activity time; the collector or query time used to establish visibility. One observed ID matching the expected ID passes the identity gate. Zero IDs is a telemetry problem. Two IDs for one intended conversation is fragmentation, even if both fragments have individually valid spans. Reusing the same display friendly ID in a different project is also a different evidence scope. Do not begin by blaming the evaluator when a trace is absent. Opik's pinned SDK configuration guide documents batching in the TypeScript SDK and explicit client.flush() and flushAll() controls. A completed flush is useful delivery evidence, but it still does not prove that the collector accepted the batch or that the query is reading the intended project. Confirm visibility after the flush boundary. This order prevents a common diagnostic mistake: Treat cooldown and sampling as eligibility, not failure Thread level online evaluation is intentionally asynchronous. Opik documents a default 15 minute cooldown after the last activity before a thread is scored. The value can be changed in workspace settings or through the documented self hosted environment setting. The same pinned documentation explains that the delay is intended to let the full conversation settle. Therefore, now last activity at < configured cooldown is active , not overdue. The agent may be working, waiting for a legitimate user turn, or simply inside the observation window. Paging at minute five when the recorded policy is 15 minutes manufactures an incident. Sampling creates a second non failure path. An Opik online rule has an explicit sampling rate alongside its model, prompt, variable mapping, and score definition. If a receipt says that a thread was not selected, the correct state is coverage excluded . It is not scoring overdue . For selected threads, add a separate scoring grace period after cooldown. That grace period is your operating SLO, not an Opik guarantee: Between those times, keep the verdict scoring pending . After overdue at , inspect rule logs, evaluator credentials, model availability, rate limits, and queue health. This creates a clean alert boundary without confusing legitimate activity with a failed evaluator. The receipt needs to preserve the policy that produced the decision. Store the actual configured cooldown, rule version, sampling decision, evaluator name, and grace period with the classification. If the cooldown changes from 15 to 30 minutes, historical events should remain explainable rather than silently acquiring a new meaning. A visible score can still be stale New activity changes the evidence version. Opik's conversation thread documentation says that adding a trace preserves existing feedback scores, restarts the cooldown, and reruns online evaluation after the new cooldown. The preservation is useful for continuity, but it creates a temporary stale green risk. Use this rule: If the visible score predates the newest turn, classify it as score stale regardless of its value. Wait for the rerun or explicitly evaluate the latest thread revision. Do not average the old score into green and do not erase it; retain it as evidence about an earlier thread state. Freshness is necessary but not sufficient. Opik stores online evaluation outputs as feedback scores, and its thread rules can judge an entire conversation. The pinned rules documentation also describes conversation coherence, user frustration, and custom metrics, including access to the execution path when the selected model supports tool calling. Those are evaluation results. They answer the question encoded in the metric. They do not automatically prove that an external side effect or deliverable exists. Suppose a support agent receives a high relevance and coherence score after saying that it updated a ticket. The thread evidence can support “the conversation was coherent” and perhaps “the expected tool call appeared.” Only the ticket system can prove that the intended ticket now contains the intended bounded change. The final gate must query that destination using a non sensitive correlation key and compare the result with a deterministic acceptance rule. This produces three distinct decisions: fresh score below threshold: quality alert ; fresh acceptable score without a destination receipt: outcome unverified ; fresh acceptable score plus a matching destination receipt: verified . The ordering is deliberate. A destination receipt does not make a poor conversation healthy, and a good conversation score does not create the destination result. Replay the ten state audit The accompanying opik thread score audit.mjs fixture contains no conversation content. Each case provides only visibility, expected and observed identity, last activity, sampling selection, score and score time, and a Boolean destination receipt. The example policy uses the documented 900 second default cooldown, a locally chosen 300 second scoring grace, and a demonstration threshold of 0.7 . The state precedence is easiest to apply as a mobile safe decision list: 1. telemetry missing : the intended trace is not visible. Check flush, collector, project, and query freshness. 2. thread fragmented : the observed IDs do not equal one expected ID. Repair propagation before judging. 3. active : the latest activity is inside cooldown. Leave it alone. 4. coverage excluded : the eligible thread was not sampled. Record coverage; do not page. 5. scoring pending : the selected thread is eligible but inside grace. Retain unknown and wait. 6. scoring overdue : the selected thread is beyond grace without a score. Inspect the evaluator path. 7. score stale : the score time predates the latest activity. Evaluate the latest revision. 8. quality alert : a fresh score is below the chosen threshold. Review evidence with bounded authority. 9. outcome unverified : the score is fresh and acceptable, but there is no destination receipt. Verify the real result. 10. verified : identity, timing, score, and outcome all pass. Retain receipts. Run the artifact from its directory: The fixed replay returns ten different expected states and exits nonzero if any case changes unexpectedly. Two cases are worth comparing: The first has a score of 0.94 , but the score predates the latest trace. The second has a fresh 0.91 , but no destination receipt. Only the third has a stable thread, completed current evaluation, an acceptable score, and verified output. Adapt the fixture by replacing the synthetic receipts with a content minimized export from your environment. Hash identifiers if equality is all you need. Keep prompt text, responses, tool payloads, secrets, and absolute local paths out of the health stream. Set the scoring grace and quality threshold from your own evaluator latency and calibration data; neither value is supplied as a universal Opik default. Use one calm operating rule For Opik LLM observability, the practical rule is: Do not interpret a thread score until the intended traces form one current thread and the scoring policy says that thread was eligible. Do not clear the run until the score is newer than the last activity and the requested result is independently verified. This rule preserves legitimate waiting, makes sampling visible, and prevents both missing score alarms and stale score green states. It also keeps the boundary honest: Opik supplies valuable trace and evaluation evidence; your destination supplies the outcome receipt. The audit has limits. It does not test evaluator calibration, prompt quality, semantic correctness, provider completeness, or the availability of a live Opik deployment. A content free state machine cannot decide whether 0.7 is the right threshold for your task. Calibrate judges against deterministic and human labels, record uncertainty, and retain a human review path for consequential decisions. Sidewisp is currently in private preview. Its planned health layer is intended to put evidence, freshness, waiting states, and verified outcomes into one operator view, but this article does not imply that an Opik adapter or production monitoring engine is shipped today. Primary sources Opik conversation and thread identity documentation, pinned to the reviewed commit Opik online evaluation rules, pinned to the reviewed commit Opik SDK configuration and flush controls, pinned to the reviewed commit