Worker Leases And Recovery

Summary

Worker leases make job execution recoverable by recording who claimed work, when that worker last reported, and when the claim should expire.

Problem

A worker can crash, a container can disappear, or a run can get stuck. Without leases and recovery, the platform cannot distinguish active work from abandoned work.

Principle

Persist execution state and recover stale claims explicitly. Liveness belongs in worker heartbeat and lease fields; business progress belongs in structured run payloads.

Platform Shape

The job execution modes doc describes worker polling, Cloud Tasks HTTP execution, stale-run recovery, and ops.job_run progress. Worker code lives under app/worker, job-run persistence lives under services/ops/execution_state/jobs, and stale recovery has an explicit ops job.

Source Evidence

• docs/reference/job-execution-modes.md
• app/worker
• services/ops/execution_state/jobs
• jobs/ops/recover_stale_runs/run.py

Related Reading

• docs/engineering/concepts/job-run-state.md
• docs/engineering/concepts/run-artifacts-and-observability.md