PostHog LLM Observability: Test the Join Key

PostHog LLM observability needs a deliberate join key once agent work can retry, leave a browser session, or share a session with another task. $session id , $ai session id , and $ai trace id are useful, but none is automatically the identity of the accepted work. In a 14 event experiment, grouping by the frontend session created two conflated groups and two wrong generation to outcome pairings. Grouping by the AI session split one retried task across two sessions and still produced one wrong pairing. A durable work id linked all three expected verified outcomes, preserved the retry as one task, and isolated a wrong work receipt as an orphan instead of attaching it to a healthy trace. The operating rule is specific: use PostHog sessions for navigation and aggregation, traces for causal model activity, and a privacy safe work id for the unit that must eventually produce an outcome. Then query those fields together. This is how traces, costs, and product analytics become an evidence contract rather than a loose collection of dashboards. Four identifiers answer four different questions PostHog's AI trace model requires $ai trace id for AI Observability events. A trace groups related generations and spans. It answers: which model and tool activity belonged to this interaction? The AI session guide defines $ai session id as an optional, application chosen grouping across traces. It can represent a workflow, thread, conversation, or another logical boundary. The same guide distinguishes it from the standard frontend $session id , which is usually captured in the browser. PostHog also uses distinct id to associate events with a person or service identity. That answers who or what emitted the event; it should not be overloaded with a task identifier. An accepted unit of agent work needs a fourth identity: Identifier Good boundary Failure when used as the work key distinct id Person, account, or service One actor can own many concurrent tasks $session id Frontend visit Background work can outlive it; one visit can start several tasks $ai session id Application defined AI session A retry or restart can create another session $ai trace id One causal trace Multi trace work fragments across retries and handoffs work id One accepted task and its outcome Must be created and propagated by the application Create work id when the system accepts the task, before the first model call. Make it opaque and stable. It should survive a retry, worker restart, approval wait, browser close, and model change. Do not derive it from an email address, prompt, path, or destination name. PostHog's generation documentation defines the generation event model. Its custom properties documentation shows JavaScript wrapper examples using posthogProperties and posthogDistinctId , while the session guide documents $ai session id as an application chosen grouping. The package version observed through the npm latest tag was @posthog/ai 8.4.0 on July 27, 2026; treat that as a dated snapshot and check the current documentation for your provider and installed version. Reproduce the 14 event join key experiment The fixture contains five accepted work IDs and one deliberately wrong outcome ID. It models three failure shapes that session only dashboards often hide: 1. work 102 starts in browser session browser b , retries after the frontend context is gone, and continues under a new AI session. Its verified outcome arrives with work id but no session IDs. 2. work 103 and work 104 start inside the same browser session. Only work 103 has a verified outcome, while work 104 has a product event but no outcome. 3. work 105 completes under AI session ai run d , but a later outcome event carries work 999 while retaining the same frontend and AI session values. Those are not synthetic naming tricks. They represent common topology changes: a background retry, concurrent tasks from one visit, and an event whose correlation metadata disagrees. I loaded the PostHog shaped events into an in memory SQL table and evaluated three strategies. A strategy receives credit only when a group contains both a generation and the expected outcome for the same work. It records a wrong pair when a generation for one work ID shares a group with an outcome for another. The measured result was: Correlation strategy Correct verified work Missed expected work Conflated groups Wrong pairs Fragmented retries : : : : : Frontend $session id 2 of 3 1 2 2 0 $ai session id 2 of 3 1 1 1 1 Durable work id 3 of 3 0 0 0 0 The frontend session join merged work 103 with work 104 and merged work 105 with the wrong work 999 outcome. The AI session join avoided the concurrent browser collision, but split work 102 across ai run b1 and ai run b2 ; its outcome had no AI session to attach. It also joined work 105 to work 999 because both carried ai run d . The work id query produced six rows: five accepted work units plus work 999 . That sixth row had one outcome and zero generations. Instead of turning work 105 green, the query exposed an orphan outcome event. Build the work matrix in HogQL PostHog documents SQL access as HogQL, a wrapper around ClickHouse SQL with simplified event property access. Event properties use dot notation, including dollar prefixed PostHog properties. Supported aggregations include countIf , uniqExactIf , and groupUniqArray . This query creates one row per durable work key: The PostHog SQL guide shows the events table, property access, SQL insights, and the HogQLQuery API shape. The aggregation reference lists the conditional and exact uniqueness functions used here. Interpret the row shape before calculating a score: generation count = 0 and outcome count 0 is an orphan outcome, not verified work. ai session count 1 can be a legitimate retry or handoff; inspect retry count before calling it a duplicate. frontend session count = 0 is normal for background work. product event count 0 shows product behavior, not destination verification. trace count 1 can be expected when one accepted task spans retries. For work 102 , the matrix reports two traces, two AI sessions, one retry, and one outcome. The row stays intact because the work key survived both session changes. That is the central result of the experiment. Audit collisions before trusting a dashboard A work matrix shows what grouped successfully. A collision audit asks whether the alternative keys would have grouped unrelated work. Run this against frontend sessions: Repeat it with $ai session id . In the fixture, the frontend audit returns browser c with work 103 and work 104 , plus browser d with work 105 and work 999 . The AI session audit returns ai run d with work 105 and work 999 . This does not prove which event is wrong. It identifies a boundary where session based attribution is unsafe and gives the operator a small investigation set. Add a second check in the other direction: count the number of distinct session values per work id . A work key with two AI sessions and a retry event is likely continuity across attempts. A work key appearing in many sessions without a retry, handoff, or resume record may indicate key reuse. The fixture and runner are deliberately inspectable. The local runner executes a SQL work matrix over all 14 events, then evaluates the three join strategies and asserts five findings: It is not a live PostHog benchmark. It does not measure ingestion latency, query API permissions, retention, or tenant specific property types. It tests the relational claim behind the dashboard. Before using the query in production, run it as a SQL insight on a harmless canary set and compare the returned columns with your fixture. Instrument the key without leaking task content Attach the same opaque work key to every relevant event. In the supported JavaScript wrapper examples, the custom properties page documents posthogProperties and posthogDistinctId , the sessions page places $ai session id inside posthogProperties , and the privacy page documents posthogPrivacyMode . Combining those documented options, the request shape looks like: Use the exact options supported by your provider integration and installed version. PostHog's privacy mode excludes $ai input and $ai output choices ; it does not sanitize arbitrary custom properties. Maintain an allowlist. Good fields are opaque IDs, attempt numbers, workflow versions, low cardinality states, timestamps, and hashes. Bad fields are prompts, completions, secrets, emails, raw file paths, and provider payloads. Emit application events with the same work id only after their underlying fact exists. A report view opened event belongs to product analytics. An agent outcome verified event should follow an authoritative read back and include a hashed destination reference plus the verified content or version hash. The two events can share a query without pretending they mean the same thing. Keep distinct id stable for the actor you want to analyze. Keep $session id and $ai session id for their documented navigation boundaries. The model becomes easier to debug because no field is doing three jobs. Use the experiment as an operating test Start with three canaries rather than a large dashboard: one task that begins and completes in one browser and AI session; one task that retries under a new AI session after the browser session is gone; two tasks started from the same browser session, with an outcome for only one. Add one deliberately mismatched outcome event in a test environment. Your work matrix should surface it as an orphan. The session collision queries should flag the shared groups. If a dashboard makes the unmatched task look verified, the join key is still wrong. Monitor the contract itself: count AI events missing work id ; count outcome events with no generation row; count accepted work IDs split across sessions without retry or handoff evidence; measure event ingestion delay before treating a recent absence as failure; alert on sudden increases in session collisions or reused work keys. Costs then become safer to interpret. Sum generation cost by work id , not merely by session, and divide only by work with the outcome status your business accepts. The result is cost per verified unit of work across retries, not cost per trace or browser visit. Where Sidewisp fits PostHog is well suited to event capture, product analysis, SQL insights, and investigation. The experiment here keeps those strengths while making the unit of operational judgment explicit. Sidewisp is intended to become a health layer around existing agent runtimes, using evidence, freshness, uncertainty, approval boundaries, and verification. It is not a replacement runtime, mandatory model gateway, or PostHog substitute. Production monitoring adapters and recovery are not shipped today. Sidewisp is currently in private preview. If this join key problem matches your environment, join the private preview waitlist and describe which runtimes, session boundaries, and outcomes your work crosses. Until then, keep PostHog's identifiers honest: sessions navigate, traces explain activity, and the durable work key carries the operational result.