Operations And Production

Summary

Operations and production is the support surface for keeping the platform deployable, observable, recoverable, and safe to change. It spans alerting, smoke checks, rollback, job diagnostics, environment bootstrap, repo guardrails, asset inventory, morning briefs, and the tests that keep common failure modes cheap to catch.

This thread starts from the support workflow rather than the source tree. When production looks unhealthy, the platform should help an engineer answer: what changed, what is failing, what state is durable, what can be retried, what requires rollback, and which domain owner needs to repair business state?

Module At A Glance

Reader Question

How does LKCI operate the platform in production without relying on tribal memory, live database poking, or one-off scripts as the primary diagnostic surface?

Surface Or Workflow

The entry points are ops diagnostics, alert investigations, deploy smoke checks, rollback runbooks, job run detail, schedule state, worker logs, asset inventory jobs, and morning brief/notification outputs. The user may be an engineer on call, a domain owner, or an operator trying to understand why an expected automated result did not appear.

Production support should make state visible before asking someone to mutate state. A diagnostic should point to job runs, queue age, stale leases, failed provider calls, data freshness, run artifacts, or domain issues. A fix should prefer retry, resume, repair action, or rollback over manual table edits.

Architecture Rationale

Production support should reveal durable state before asking someone to mutate state. Retries, leases, run artifacts, diagnostics, escalation, rollback, and post-incident hardening are platform behavior, not tribal memory.

Lifecycle

The lifecycle starts before deployment. Repo hooks, tests, docs-path checks, and architecture guardrails catch violations while changes are still local. Environment bootstrap keeps config-dependent commands explicit. Deployment then needs smoke checks, readiness signals, and rollback procedures.

In production, alerts and diagnostics route attention. Queue depth, worker retries, stale recovery, API errors, and readiness failures tell engineers which platform layer is under stress. From there, ops state and job artifacts show what ran, what failed, what can be retried, and what must escalate to a domain review surface.

After repair, the system should leave evidence: state transitions, artifacts, logs, tests added, runbook updates, or guardrail improvements. Production operations is therefore not just incident response; it is the loop that turns recurring failure into platform architecture.

Child Threads

  • Operations Guardrails Testing: how hooks, tests, config bootstrap, IaC, deployment gates, and runbooks protect the platform.
  • Operations Production State Machine: the system-level lifecycle from trigger through lease, execution, retry, block, escalation, recovery, and audit.
  • Ops Diagnostics: signals, thresholds, ownership, and escalation.
  • Environment Bootstrap Contract: explicit config loading for local and operational commands.
  • Repo Root And Package Root Guardrails: structure checks that keep the repo legible.
  • Asset Inventory Jobs: source-backed operational inventories.
  • Morning Brief And Notifications: scheduled summaries and communication.

Implementation Boundaries

services/ops owns support state and observability helpers. jobs/ops owns thin operational entrypoints. infra/hooks encode architectural rules. config owns environment bootstrap. ops owns deployment and IaC assets. Runbooks under docs/runbooks remain the procedural source of truth; this thread explains how those procedures fit the platform model.

Tradeoffs

Operational rigor can feel slow when a local script would produce a quick answer. The platform deliberately favors repeatable diagnostics, explicit state, and documented rollback because production support has to survive handoffs. The best operations content should keep engineers from rediscovering the same support path during the next incident.

Visual

The current visual is a graph neighborhood. The intended first-level visual is a production support map from change gate to deploy, smoke, alert, diagnostic, retry/repair, rollback, and post-incident hardening.

Operating Posture

Production operations starts before a service is deployed. The repo's local gates, changed-file quality checks, docs-path checks, package-root checks, boundary hooks, and size reports are part of the operating system. They keep the common failure modes cheap enough to catch while a developer still has the change in context. That matters because the platform is broad: data products, Cloud Run services, jobs, DAGs, UI surfaces, provider integrations, and docs all change in the same repository.

The changed-quality gate is deliberately pragmatic. It does not pretend that every local edit needs the full test universe, but it does make meaningful changes pass the checks most likely to catch architectural drift. The broader CI and predeploy gates then layer on the slower assurances: web builds, Terraform validation, production plans, and domain-specific tests. The operations posture is therefore staged. Fast checks protect momentum; fuller checks protect deployment.

Environment bootstrap is part of the same posture. Commands that depend on configuration, credentials, graph state, or provider access should load their environment explicitly. That rule makes local debugging, scheduled jobs, and production reproduction less dependent on hidden shell state. A job or probe that only works because one terminal happens to have the right variables is not operationally durable.

Production Support Flow

When production looks unhealthy, the first question should be "which durable state changed?" A failed request, missing artifact, stale dashboard, or undelivered message is a symptom. The platform should lead support toward job runs, DAG runs, queue state, leases, source freshness, provider outcomes, issues, review cases, logs, and run artifacts. The purpose of the ops surface is to make those signals easy to correlate.

The second question is "what can be retried safely?" Retryability is not a generic property of an exception. It depends on idempotency, provider state, command state, artifact writes, and domain semantics. A marketing render step, a POA command, a precon document download, a payroll sync request, and a morning brief email all need different retry evidence. Operations should not hide those differences behind a single "rerun everything" button.

The third question is "who owns the repair?" Some failures are platform failures: bad deploys, stale workers, broken config, missing secrets, queue misconfiguration, or provider outages. Others are domain failures: ambiguous POA matching, incomplete precon documents, payroll blockers, invalid campaign membership, or notification payload validation. The ops layer should route the reader to the right owner instead of letting support collapse into ad hoc data edits.

Rollback And Hardening

Rollback is an operational behavior, not an admission that the platform failed. The deploy smoke and rollback runbook exists because production changes should have a known recovery path before they are needed. The same idea applies to data publication and job rollout. A bad code deploy, stale data product, or misconfigured dispatch path should have a documented way to pause, revert, retry, or route work back through a safer mode.

After recovery, the platform should collect the lesson. That might mean a new test, a stronger hook, a clearer runbook, a diagnostic endpoint, a better artifact, or a domain-specific repair state. The point is not to eliminate all incidents. The point is to make repeated incidents more structured each time they occur. Production operations is therefore one of the main feedback loops for the architecture.

Relationship To Domain Systems

Operations does not replace domain ownership. It gives each domain enough shared infrastructure to be supportable. Marketing owns campaign lifecycle and review. Geo owns source and scoring jobs. POA owns procurement document state. Precon owns opportunity and document evidence. Payroll owns review and sync state. Notifications owns delivery semantics. Operations owns the cross-domain patterns: deploy, smoke, alert, job health, diagnostics, rollback, and guardrails.

This division keeps support conversations precise. If a provider send failed, the provider boundary and notification diagnostics should explain the outcome. If a precon sizing job produced incomplete evidence, the precon repair path should own the business response. If a Cloud Run worker cannot reach a queue, operations owns the runtime issue. The thread exists so those boundaries are visible before a production incident forces someone to rediscover them.

Source Evidence

  • services/ops
  • jobs/ops
  • docs/reference/testing.md
  • docs/reference/operations-alerting-baseline.md
  • docs/runbooks/deploy-smoke-rollback-runbook.md
  • infra/hooks
  • config
  • ops