AI Agent Observability: Test Decision Continuity After Compaction

Context compaction should be treated as an observable continuity boundary, not a routine “session still alive” event. The practical default for AI agent observability is to capture a small set of task decisions before compaction, compare them after the agent resumes, and keep the run unhealthy or uncertain if a decision disappeared, changed without authority, or lacks fresh evidence. The invariant set does not need the conversation. For consequential work, it can contain three to seven identifiers such as scope , approval boundary , target , and acceptance test , plus a digest, revision, observation time, and approval reference. A matching digest means the decision survived. A changed digest is healthy only when an explicit human approved transition connects the old and new values. That extra test closes a gap left by activity telemetry. An agent can emit a post compaction event, reload project instructions, call tools successfully, and make visible progress while silently working toward the wrong branch, skipping a required approval, or using an obsolete definition of done. Observe the boundary without calling it healthy Different runtimes expose compaction differently. Claude Code is a useful concrete example because its current context window documentation states that /compact replaces the conversation with a structured summary. Startup context generally reloads, while the verbatim conversation, full tool output, and intermediate reasoning are no longer present. The same documentation also describes budget limits and exceptions for skill reinjection. Its hooks reference exposes several relevant lifecycle signals: PreCompact fires before manual or automatic compaction; PostCompact fires after compaction finishes; SessionStart can report source: "compact" ; InstructionsLoaded can report reason: "compact" . Those events establish that a boundary occurred and that execution continued. They cannot prove that the compact summary retained every task decision. InstructionsLoaded also answers a narrower question: an instruction source entered context. It does not prove that the agent interpreted it correctly or that a task specific choice from 40 turns ago was present in that file. The distinction between durable instructions and current decisions matters. Claude Code’s memory documentation describes CLAUDE.md and auto memory as persistent context, while explicitly noting that they are context rather than enforced configuration. Project conventions belong there. A temporary branch choice, a one run permission boundary, or the acceptance test negotiated for the current task belongs in a separate task record. Do not alert merely because automatic compaction occurred. Compaction is normal context management. Alert when the evidence on its two sides disagrees, or when the post compaction observation never arrives within a runtime specific window. Define a decision invariant, not a transcript archive A continuity record should be small enough to review and safe enough to retain. This example stores digests rather than raw decision values: The names are deliberate: scope identifies the repository, service, data set, or destination that is in bounds; approval boundary records which action still requires human authority; target can identify the branch, environment, issue, or artifact expected; acceptance test identifies the deterministic check that resolves the task; constraint covers a task specific prohibition that must survive the summary. Normalize a decision before digesting it. Sort object keys, use one character encoding, distinguish null from a missing field, and version the normalizer. Otherwise, harmless formatting changes look like drift. Keep the pre compaction snapshot outside the conversation being summarized; a local append only file or task database is enough. Plain hashes are not encryption. A digest of a predictable secret or a two value policy can be guessed by enumeration. Never place secrets, prompts, responses, raw tool payloads, transcript paths, or absolute private paths in this ledger. For sensitive low entropy values, store an opaque identifier or use a keyed HMAC held outside the agent context. An approved change needs more than a new digest: That record distinguishes legitimate adaptation from decision drift. The monitor should not reject every change; it should reject unexplained change. Reproduce the six case continuity test I tested an event only rule against an invariant aware classifier using six synthetic compaction boundaries. Every case contained a completed pre/post event pair, so the event only rule marked all six healthy. The stronger rule compared every pre compaction decision with its post compaction counterpart: Run the retained artifact from its directory: The resulting counts are: Two cases retained all three decisions exactly. One changed its acceptance test through an explicit human approval. The remaining three lost an approval boundary, changed scope without authority, or repeated a test digest without a fresh observation time. This result does not estimate a universal failure rate. The fixture is a boundary test, not production telemetry. Its falsifiable claim is narrower: an observed compaction pair cannot distinguish those six states, while the invariant comparison can. Route five states to different actions The state should determine the smallest safe response. State Meaning Default action preserved every required decision matches with fresh evidence continue and retain the receipt approved change a decision changed through an explicit authorized transition continue with the new revision and audit link dropped a required decision is absent after compaction pause consequential work and restore or request the missing decision conflicting a digest changed without a matching approval stop at the next reversible boundary and ask a human unverified the value is present but freshness or source evidence is missing re observe; do not infer health from model prose Priority matters when several defects occur together. A missing approval boundary should outrank a stale acceptance test observation because it changes what the agent is allowed to do. A scope conflict should stop writes even if tool activity and tests look healthy. By contrast, an approved change is not an incident merely because the hash differs. Keep activity and continuity as separate health dimensions. A run can be working and conflicting at the same time: useful output is moving, but toward an unauthorized target. It can also be waiting human and preserved : all decisions survived, but the next action still needs approval. Flattening those combinations into one green or red light removes the operator decision the telemetry was meant to support. Implement the check with honest limits Start with one consequential task type and three invariants. Capture the pre boundary snapshot when the runtime exposes a compaction warning. Capture the post boundary snapshot after startup instructions have reloaded and before the next consequential write. Compare normalized digests locally, emit the state with timestamps, and retain the approval reference for any revision. Do not make PreCompact a universal blocking gate. A failed hook can consume the remaining context or strand normal work. If the pre snapshot cannot be written, record continuity evidence missing and restrict consequential actions until a safe re observation is possible. The correct fallback is uncertainty, not a fabricated match. The invariant set is also incomplete by design. Hash equality proves only that the recorded decisions match. It cannot reveal an important decision that nobody chose to record, a compromised ledger, or a bad acceptance test that survived perfectly. Review the set when incidents expose a missing boundary, and verify the final outcome independently of the agent’s summary. Sidewisp is currently in private preview. Its public site and article library are live, but production agent health collection, context monitoring adapters, and recovery are not generally shipped. The intended product direction is a health layer alongside existing runtimes, with evidence, visible uncertainty, and human authority—not a replacement runtime or autonomous fixer. For now, the operating rule is usable without Sidewisp: observe the compaction event, compare a minimal decision invariant, accept only explicit revisions, and let missing evidence remain unknown.