AI Agent Platform: A Six-Failure Buying Test

An AI agent management platform is worth adopting when it closes an operating gap that your current runtime cannot close safely. For a small team, the default is not “move every agent into one control plane.” Keep the runtime that already executes the work, then test whether you are missing four separate planes: execution, operational health, authority, and evaluation. That distinction matters because the category label is unusually elastic. Current vendor pages use “agent management” for catalogs, traffic proxies, policy enforcement, cross platform inventory, tracing, business KPIs, evaluation, and lifecycle control. Those are valid capabilities, but they do not answer the same question. A platform can trace every tool call and still miss a promised deliverable. It can inventory every agent and still restart one that is correctly waiting for approval. Use a six failure proof before buying or migrating. The candidate stack must detect a missed scheduled run, reject false success, preserve a legitimate approval wait, contain tool permissions, stop a costly retry loop, and catch a quality regression. Require inspectable evidence and a bounded response for each case. A feature list is only an input to that test. Separate the four planes before comparing products The first buying artifact should be a responsibility map, not a vendor spreadsheet. Plane It owns Evidence that it works It must not be allowed to claim Execution starting, scheduling, pausing, cancelling, and resuming work stable run identity, scheduler receipt, runtime state, exit reason that the intended outcome exists Operational health reachability, useful progress, waiting, outcomes, freshness, and cost anomalies heartbeat age, progress delta, typed dependency, outcome predicate, evidence timestamp authority to perform an irreversible fix Authority and governance identity, tool scope, policy, approval, audit, and action limits scoped credential, normalized action preview, bound approval, policy version, audit record that a permitted action was useful Evaluation output quality across a versioned set of cases dataset version, rubric or deterministic test, calibration record, regression threshold that a live run is reachable or currently stuck A product may cover one plane or all four. That is not a quality judgment. It tells you what must be integrated and which claims need a separate verifier. The current US search results illustrate the category spread. Gravitee’s agent management page describes an enterprise control plane built around catalogs plus LLM, MCP, and A2A proxies, policy enforcement, lineage, and traffic visibility. AvePoint’s AgentPulse page emphasizes centralized discovery, governance, lifecycle control, and visibility across platforms. Dataiku’s agent management page emphasizes a cross platform control tower, business KPIs, evaluation, drift, and governance; the same page says availability is planned for September 2026. Those are descriptions from the vendors’ own pages, reviewed July 25, 2026—not independent performance tests. More importantly, they describe different centers of gravity. “Supports management” is therefore too vague for a procurement requirement. Start with the failure you cannot diagnose today Write one sentence naming the operational failure that justifies change. “We need one dashboard” is not a failure. These are: A scheduled research agent sometimes fails to start, and the team notices a day later. A coding agent says it is done although the required test never passed. A tool using agent is quiet because it needs approval, but the operator mistakes the wait for a stall. A retry loop consumes time or tokens without changing the deliverable. A shared credential lets an agent write outside the intended project. A new prompt version passes live health checks while output quality regresses. Now assign the failure to a plane. A missed start belongs first to execution: you need the expected schedule occurrence, a stable slot or run identifier, and a start deadline. False success belongs to health: compare declared complete with an external outcome predicate. An approval wait spans health and authority: the health record must say waiting , while the approval record binds a person’s decision to an exact action. Quality regression belongs to evaluation, not to the liveness monitor. This prevents a common category error. OpenTelemetry’s GenAI agent semantic conventions define development status spans for creating and invoking agents, invoking workflows, planning, and executing tools. That is useful trace evidence. It does not define your report, pull request, ticket, or customer update as complete. Add that predicate in the application or health plane. The reverse error is just as expensive. An outcome verifier can prove that a report exists; it cannot explain whether a long running agent is working, legitimately waiting, unreachable, or looping. Keep the clocks separate: One timestamp cannot safely replace the other three. Put six failures through the same buying test The retained artifact for this article turns the four plane map into six executable cases. Each case lists the capabilities required to diagnose it and choose a bounded response: Save the full fixture as platform buying test.json , then run: The exact retained output is: The result does not prove that every runtime or tracing product has those scores. These are deliberately synthetic stack definitions. The falsifiable claim is narrower: a capability checklist passes a failure only when it contains every evidence or control primitive declared for that case. Edit the requirements to match your workflow and the result must change. That makes the artifact useful in a vendor call. Ask the candidate to show the same six cases with your own run data. Do not accept a screenshot of a normal run as proof that false success or approval waiting is handled correctly. Inspect evidence, action, and absence For each case, require three things on one page: 1. Evidence: What observation produced the state, when was it collected, and which adapter or rule version interpreted it? 2. Action boundary: What can the system do automatically, what requires approval, and what is forbidden? 3. Absence semantics: Does a missing signal mean healthy, failed, or unavailable? The third question catches many polished demos. If the outcome collector stops reporting, a green status is fabricated certainty. If a trace is sampled out, absence of an error span is not proof of success. If an agent does not expose a typed wait reason, the system may need to report uncertain instead of restarting it. Authority needs the same precision. The OWASP AI Agent Security Cheat Sheet recommends least privilege tools, explicit authorization for sensitive operations, action previews, audit trails, bounded autonomy, rate limits, and approval records tied to the exact actor, tool, target, parameters, timestamp, and expiry. That is stronger than a generic “human in the loop” checkbox. Use a compact proof record: Do not put secrets, raw credentials, complete prompts, or unrestricted tool payloads in this record. The health view needs enough evidence to support a decision, not a second copy of every sensitive input. Choose the smallest architecture that passes There are three reasonable outcomes from the test. Keep the runtime and add a health layer when execution already works, but you cannot distinguish progress from activity, waiting from stuck, or command completion from the intended result. This is often the least disruptive option for a solo founder or small engineering team. It preserves scheduler and runtime semantics while adding a normalized health record. Add a governance or traffic layer when the urgent gap is identity, tool scope, policy enforcement, registry, or cross agent traffic control. A proxy can be valuable when many agents share models, MCP servers, APIs, or A2A connections. It should not be asked to invent a task specific outcome that only the application can verify. Adopt a unified management platform when cross runtime inventory, policy, lifecycle control, monitoring, evaluation, and delegated ownership must be operated together—and the integration cost is lower than maintaining the seams. Require an export path for run identity, evidence, approvals, and outcome records so the decision remains reversible. Migration is not free. A platform adapter may flatten a runtime’s native states; scheduler occurrences can lose their identity; sampled traces can hide rare failures; a centralized proxy can become a new availability dependency. Pilot one consequential workflow before moving the fleet. The default recovery policy should also survive the move: working: leave it alone; waiting: route the dependency once; stuck: prepare one reversible retry within time and cost limits; false success: reopen the task against the failed predicate; unreachable or uncertain: collect missing evidence before changing the agent; high impact action: require explicit, action bound authority. Command completion is never enough to clear an incident. Verify useful progress or the promised result after the action. Run a proof of fit, not a feature tour Give each candidate the same two hour exercise. Use one real workflow, a sanitized copy of its evidence, and six fixtures: normal completion, missed start, approval wait, retry loop, permission violation, and quality regression. In the first hour, ask the candidate to ingest or correlate the evidence. Record exactly which native state was preserved, which was inferred, the collection delay, what was sampled, and what sensitive content crossed a boundary. Force one collector offline and inspect whether the status becomes unavailable. In the second hour, ask an operator unfamiliar with the setup to diagnose the six cases. The operator should be able to name the impact, evidence freshness, confidence, and next safe action without reading raw conversation history. Trigger one bounded response, then verify the expected state rather than merely checking that the command ran. Reject the candidate for this workflow if it cannot: retain stable identity across a retry or resume; represent waiting separately from stuck; attach a deterministic outcome check where one exists; show unavailable evidence as unavailable; bind approval to the exact action; limit retries by count, time, and cost; preserve an audit trail and an export path. The trade off is that this test favors operational correctness over breadth. It does not benchmark query speed, support quality, total cost, evaluator accuracy, or every security control. Those need separate tests. It does stop a broad category label from deciding your architecture. Sidewisp’s product direction is the operational health plane around existing agent runtimes: evidence, freshness, useful progress, waiting states, verified outcomes, prioritized issues, and explicit approval boundaries. It is not intended to be a replacement runtime, mandatory gateway, generic workflow engine, or enterprise control plane. Sidewisp is currently in private preview. The public site and article system are live, while production agent health collection, runtime adapters, cron management, token cost analytics, and recovery execution are not generally shipped. Join the private preview if the six failure test matches the operating gap you need to close. Primary references OpenTelemetry: Semantic Conventions for GenAI agent and framework spans — development status agent, workflow, plan, and tool span definitions; reviewed July 25, 2026. OWASP: AI Agent Security Cheat Sheet — least privilege, human approval, action integrity, audit, rate limits, and monitoring guidance; reviewed July 25, 2026. Gravitee: AI Agent Management Platform — official product description used to inspect the category’s proxy, policy, catalog, and lineage scope; reviewed July 25, 2026. Dataiku: Agent Management — official product description used to inspect control tower, KPI, evaluation, governance, and stated availability scope; reviewed July 25, 2026. AvePoint: AgentPulse — official product description used to inspect discovery, governance, lifecycle, and visibility scope; reviewed July 25, 2026.