Agent Monitoring for Scheduled Work: Build an Expected-Run Envelope

Agent monitoring for scheduled work should begin with one question: did this specific scheduled occurrence start, finish, and produce the promised result inside its allowed window? A green process exit, a recent heartbeat, and a complete trace cannot answer that question alone. The practical default is an expected run envelope . For every occurrence, record the intended schedule time, a permitted start delay, a maximum runtime, and a deadline for verifying the deliverable. Keep these timestamps separate. A job may be waiting legitimately, late to start, still working, overdue, duplicated, or finished without an outcome. Collapsing those states into “running” and “failed” creates noisy alerts and hides false success. This guide builds that envelope as a runtime neutral contract. The included nine case fixture is synthetic, not production evidence, but it is executable and exposes the decisions a monitoring system must make. Anchor the record to the scheduled occurrence Do not infer the expected time from the first log line. Obtain the scheduler’s intended occurrence time and retain it as scheduled at . Kubernetes 1.32 and later adds batch.kubernetes.io/cronjob scheduled timestamp to created Jobs. Google Cloud Scheduler sends X CloudScheduler ScheduleTime , which remains constant across retry attempts. Those values survive a late start and make retries attributable to the same occurrence. Use a stable slot key: Then keep these fields: outcome ref should identify evidence, not contain the sensitive deliverable. It might be a hash, an object version, a test run ID, or a database row key. A generated file merely existing may not be enough; verification should match the real promise, such as “today’s brief exists, has five cited items, and is stored at the expected destination.” The scheduler contract matters because execution is not necessarily exactly once. Kubernetes documents that a CronJob can sometimes create two Jobs or no Job and advises idempotent workloads. Cloud Scheduler describes at least once delivery and likewise requires idempotent targets. Monitoring must therefore treat duplicate starts as a first class state, not an impossible anomaly. Calculate three deadlines, not one timeout Define the envelope with three independent limits: The values should come from observed runtime distributions and business requirements, not a universal preset. A task scheduled at 09:00 may be perfectly healthy when it starts at 09:00:40. The same 40 second delay may violate a sub minute dispatch promise. Amazon EventBridge Scheduler, for example, documents 60 second invocation precision; treating second 01 as “late” would misread that scheduler’s contract. The three limits answer different questions: State Evidence Operator response waiting for start No run exists, but start deadline has not passed Wait missed start No run exists after start deadline Check scheduler and reachability running One run is active before finish deadline Leave it alone overrun The active run passed finish deadline Inspect progress before interrupting outcome pending Process finished; verification window remains open Wait for the verifier outcome missing Verification deadline passed without evidence Investigate false success duplicate start More than one run claims the same slot key Contain side effects; inspect retry cause healthy The promised outcome was verified Close the occurrence suspended An explicit maintenance or approval pause covers the slot Suppress failure; retain audit evidence This ordering prevents two common mistakes. First, absence is not failure until the applicable deadline passes. Second, process completion is not task completion. A run that exits at 09:06 can remain outcome pending until its upload, test, or destination check finishes. It becomes outcome missing only after that separate grace window expires. An overrun is also not permission to kill an agent. Check whether useful progress is still moving, whether it is waiting on an external system, and whether interruption is reversible. The envelope identifies where attention is justified; it does not make the recovery decision. Reproduce the classifier with nine awkward cases The run artifact evaluates newline delimited fixtures with a deterministic classifier. Run it with: The fixture uses a two minute start grace, a ten minute maximum runtime, and a two minute outcome grace. Its result is: The core classifier is deliberately small: This experiment demonstrates the value of explicit boundaries, but it does not prove that the chosen thresholds fit a real workload. It also assumes one scheduler occurrence maps cleanly to one slot. Event driven fan out, manually replayed historical work, and tasks with multiple required deliverables need an expanded identity model. Handle duplicates, overlap, time zones, and pauses explicitly Retries and overlap are related but not identical. A retry may repeat the same slot after a transport failure. An overlap may begin the next slot while the previous one is still active. Retain both slot key and run id , then apply the scheduler’s declared concurrency behavior. Kubernetes exposes Allow , Forbid , and Replace concurrency policies. Under Forbid , a skipped occurrence while the previous Job is active counts as missed. Under Replace , the new occurrence displaces the old Job. Your monitoring state should preserve that reason; otherwise an intentional replacement looks like a crash. For side effecting tasks, deduplicate on the slot key at the destination as well as in the monitor. A second “successful” run can still send a second invoice, overwrite a newer report, or publish the same message twice. The monitor can expose the risk, but idempotency belongs in the workload and destination contract. Time zones need an equally explicit rule. Store occurrence timestamps in UTC while retaining the schedule’s IANA time zone identifier and original expression. Daylight saving transitions are scheduler specific. EventBridge Scheduler documents that a nonexistent local time during spring forward is skipped and a repeated local time during fall back runs once. Do not synthesize a “missed” occurrence that the scheduler never promised. Finally, pauses must be modeled, not hidden by disabling alerts. Record who paused the schedule, why, the start and expiry time, and whether catch up is expected. Kubernetes notes that suspended CronJob occurrences count as missed and may run immediately after unsuspension when no starting deadline is set. A monitor that forgets the pause can flood the operator precisely when maintenance ends. Turn the envelope into a quiet operating rule Start with one critical scheduled agent, not every trace: 1. Read the scheduler’s native occurrence timestamp, time zone, retry policy, and concurrency policy. 2. Assign a slot key before work begins and preserve it through retries. 3. Choose start grace , max runtime , and outcome grace from actual requirements and observed durations. 4. Define one deterministic outcome verifier. 5. Replay recent history through the nine states before enabling notifications. 6. Page only when a user relevant promise is outside its envelope; keep waiting for start , running , and outcome pending visible but quiet. Revisit the thresholds after schedule, model, tool, or destination changes. A larger model may raise runtime without changing correctness. A slower external API may lengthen outcome verification. Threshold drift is configuration debt, not evidence that an agent became unreliable. This contract also sets a useful data boundary. You need timestamps, stable identifiers, state, and a reference to verification evidence. You do not automatically need prompts, responses, raw tool payloads, or full traces. Collect those only when a diagnosis requires them and your privacy policy permits it. Sidewisp is intended to turn signals such as missed schedules, stalls, tool failures, and missing outcomes into a prioritized health view with explicit evidence and approval boundaries. The production monitoring engine and runtime adapters are not generally shipped today. Sidewisp is currently in private preview. If this expected run contract matches a failure you operate, the private preview signup is the restrained next step—not a claim that Sidewisp already monitors your live agents. Primary sources Kubernetes CronJob documentation — scheduled timestamps, start deadlines, concurrency policy, suspension, approximate creation, and idempotency. Google Cloud Scheduler overview — at least once delivery, retry behavior, idempotency, and the stable scheduled time header. Amazon EventBridge Scheduler schedule types — invocation precision, time zones, and daylight saving behavior.