Operations Production State Machine

Summary

Production operations coordinate scheduled work, triggered work, worker leases, run state, issues, retries, escalation, recovery, and audit. The state machine keeps automation observable and repairable.

Reader Question

How does operational work move from trigger to execution to completion, retry, or human escalation without hiding failure in logs?

State Machine

Triggered starts with a schedule, event, operator action, or dependent job. Leased assigns work to a worker or execution context. Running records progress, artifacts, and checkpoints. Succeeded publishes completion and observability facts.

Failures branch into Retryable, Blocked, or Escalated. Retryable work can resume through idempotent services. Blocked work needs data freshness, provider availability, configuration, or dependency repair. Escalated work becomes an issue or review item with enough context for a human to decide what should happen next.

This state machine is shared vocabulary for job families, DAGs, smoke checks, runbooks, and production dashboards.

Visual

The diagram shows execution state as a first-class operational artifact, not a side effect of a cron command.