Agent Memory Benchmark: Audit the Gaps Behind One Score

An agent memory benchmark is useful only when its test matches the failure you need to catch. Retrieval accuracy can show that stored material is findable. It cannot, by itself, show that memory survived a restart, that the newest version won a conflict, or that an agent used the memory to produce the intended external result. The practical default is therefore not “pick the highest score.” Pick the benchmark whose method exercises your riskiest decision, keep its uncovered questions visible, and run a small production canary beside it. I audited three current approaches—AMB, MemoryArena, and STATE Bench—against seven questions. None documents the whole operational set. Three benchmarks, three different units of evidence AMB documents a four stage pipeline: ingest documents, retrieve context, generate an answer, then use a second model to judge that answer against gold answers. It tracks retrieval and ingestion time as well as accuracy, speed, and token cost. That makes AMB useful when the decision is about retrieval quality and economics across memory providers. Its method also explains why “agentic” in a dataset name is not enough. The documented terminal event is still a generated answer and a judge result. That is meaningful outcome evidence, but it is not the same as observing an agent change a destination system correctly. MemoryArena moves the unit of evidence into a memory agent environment loop. Its 766 human crafted tasks have interdependent subtasks across sessions. Later actions depend on information and feedback acquired earlier, across web navigation, constrained planning, progressive search, and sequential reasoning. The paper reports an average of 57 action steps per task. That design addresses a real weakness in recall only evaluation: an agent can retrieve a fact yet fail to apply it when the environment changes. MemoryArena makes memory consequential to later action. A multi session task, however, is not automatically a process restart test. Unless the harness deliberately tears down the memory component and proves durable state after startup, persistence remains a separate question. STATE Bench tests 450 stateful enterprise tasks across travel, customer support, and shopping. Its agent learning track can supply reusable memories through a retrieval hook. The headline metrics are task completion pass@1, pass^5 across five runs, an LLM judged user experience score, and cost per task. The scoring boundary matters. Microsoft’s release description says state mutating tasks use deterministic assertions against the final environment state, while procedural and informational tasks use an LLM judge. “STATE Bench checks outcomes” is accurate; “every STATE Bench outcome is deterministic” is not. Keep coverage states instead of manufacturing one score I mapped each documented method to seven questions. Covered means the method directly exercises the question. Partial means it supplies relevant evidence but stops short of the full operational claim. Not documented means exactly that; it is not an accusation that a project cannot run such a test. Evidence question AMB MemoryArena STATE Bench Retrieval quality Covered Partial Partial Memory guides later action Partial Covered Covered Deterministic final outcome Partial Partial Partial Deliberate restart persistence Not documented Partial Not documented Freshness and provenance Not documented Not documented Not documented Repeated run reliability Not documented Not documented Covered Cost or efficiency Covered Not documented Covered This table should not be reduced to “two out of seven” or “five out of seven.” Coverage is not additive. If the incident you fear is a stale refund policy surviving an update, retrieval accuracy on a static corpus and five stable task runs can both be green while the dangerous conflict remains untouched. The audit artifact validates the matrix and lists every uncovered operational question without calculating a composite score: Running that rule against the source backed matrix produced: This result is falsifiable. A benchmark can change its method, add an explicit restart fixture, require versioned provenance, or replace a judge with deterministic destination assertions. Update the matrix when the public method changes. Do not preserve an old “not documented” label as folklore. Choose by the failure, not by the leaderboard Start with the concrete decision you will make after the run. If you are choosing a retrieval provider, AMB’s ingest/retrieve/generate/judge separation and timing data are directly useful. Run the same dataset, domain, generation model, judge prompt, and mode for every provider. Its README warns that small prompt or model changes can move accuracy by double digits, so a comparison without those version pins is not reproducible. If your risk is that remembered information is available but does not change later behavior, use an action coupled test. MemoryArena’s interdependent subtasks make earlier information causally necessary in later sessions. Preserve the task manifest and environment version, then inspect the first action that diverges—not only the final task score. If the decision is whether memory improves a stateful workflow consistently, STATE Bench offers a stronger default. Compare the no memory baseline with the same agent plus memory; keep pass@1, pass^5, cost, and the scorer type separate. A gain in average completion alongside a drop in pass^5 is not a clean promotion. Nor is an LLM judged procedural pass equivalent to a deterministic database assertion. These options can be combined. A small team might run AMB to eliminate a retrieval implementation that is too inaccurate or costly, then run a focused action coupled suite, then use a STATE Bench subset for repeated workflow outcomes. The sequence is defensible because each stage answers a named question. A blended leaderboard number would hide why a candidate passed. The limitation is important: this audit compares documented methods; it does not rerun every benchmark or claim that undocumented private tests do not exist. Synthetic tasks, simulated users, evaluator models, domain coverage, and repository revisions all limit how far a result transfers to your workload. Add the operational checks benchmarks leave open Offline evidence should end at a promotion boundary. A compact live canary should start there. Use content free identifiers when real memory may contain private material. For example, write a random canary ID, a version, a hash of an expected decision, and an expiry time. Keep the raw prompt, conversation, secret, and customer data out of the health event. Then run this chain: 1. Acknowledge the write. Record the memory namespace, canary ID, expected version, adapter version, and write receipt. A successful HTTP response is not proof that the required index or durable store accepted the record. 2. Cross a real restart boundary. Restart the memory service, agent runtime, or host adapter you actually depend on. Starting a new chat inside the same process tests a different boundary. 3. Read with freshness and provenance. Require the expected version and an inspectable source reference. A semantically plausible older value is a failure, not a near miss. 4. Inject a conflict. Write a newer value, retain the old identifier, and verify the resolver returns the intended winner. Record whether the policy is last write, explicit version, source priority, or human approval. 5. Require a consequential action. Give the agent a task where the correct tool argument depends on the canary. Verify the argument or state transition, not the explanation the model produces. 6. Verify the destination. Read the external system or artifact that represents completion. Command exit, tool call success, and agent reported success are activity evidence. 7. Repeat within bounds. Run enough equivalent cases to reveal inconsistency, while fixing model, prompt, tool, and environment versions. Track cost beside the verified outcome. A minimal receipt can look like this: Do not mark the memory path healthy if a required field is unavailable. Return uncertain , preserve the last known evidence time, and route the missing signal to a person. The safe response to incomplete evidence is not an automatic retry that may repeat an external effect. A promotion rule you can explain Promote a memory change only when three statements are all true: the chosen benchmark directly exercises the behavior that motivated the change; repeated results improve or hold quality without violating the agreed cost boundary; the live canary proves restart persistence, correct version/provenance, consequential use, and the expected destination outcome. If any statement is unsupported, keep the change in evaluation. This rule is deliberately stricter than “the benchmark went up,” but narrower than building a universal evaluation platform. It gives a small team an auditable reason to proceed or stop. Sidewisp’s product direction treats missing memory reads or writes, failed persistence across restarts, stale synchronization, resets, context growth, and lost decisions as health signals. The production monitoring engine and runtime adapters are not shipped in the current website repository. Sidewisp is currently in private preview. If that evidence model matches how you operate agents, you can join the private preview waitlist without replacing your runtime or treating an offline benchmark as a live health certificate.