Lakehouse And Execution

Summary

Lakehouse and execution connect two questions: what data products must stay correct, and what automated work keeps them current? This thread explains how pipelines, jobs, schedules, DAGs, leases, and run metadata support data freshness without turning pipelines into UI workflows.

Reader Question

How does a source-backed data product get refreshed, monitored, and recovered without hiding work in a monolithic script?

Surface Or Workflow

The entry surfaces are the jobs console, DAG authoring console, freshness signals, and domain workflows that depend on published facts. A reader might start from a stale insight, failed materialization, delayed sync, or scheduled job run.

Lifecycle

Execution starts with an explicit job or DAG task. Thin job entrypoints parse runtime parameters and call services, pipelines, or integrations. Scheduler and worker infrastructure records run state, artifacts, issues, and retries. Lakehouse services manage read/write boundaries so source seeding, canonicalization, and publication remain reproducible.

When execution fails, the platform should not silently drop work or require a developer to inspect logs only. It should record artifacts and issues, expose diagnostics, and either retry safely or escalate to a review/repair surface.

Child Threads

  • Automation With Human Repair: how execution escalates into repairable state.
  • Lakehouse Publication State Machine: how source capture, bronze, silver, gold, quality checks, publish, serving, stale, and failed states compose.
  • jobs-console-and-job-catalog: how operators inspect job identity, status, and parameters.
  • dag-authoring-and-task-repair: how composed execution is represented and resumed.
  • worker-leases-and-recovery: how long-running work avoids duplicate or lost execution.
  • incremental-source-seeding: how source refresh stays deterministic.
  • crosswalks-and-identity: how execution depends on stable entity links.

Implementation Boundaries

This thread is backed by docs/domains/lakehouse.md, docs/domains/dags.md, docs/reference/job-layout.md, and docs/reference/job-execution-modes.md. Implementation evidence spans jobs/lakehouse, pipelines/lakehouse, services/execution, services/workflow, and DAG-related UI/API routes.

Tradeoffs

Explicit jobs and DAGs add metadata and operational overhead. They are worth it because data correctness depends on repeatable execution, visible failures, safe retries, and a path from automated work into human repair.

Visual Notes

The target visual is an execution timeline: schedule or trigger to job/DAG run to service or pipeline call to artifact/issue to retry, publish, or repair.

Source Evidence

  • docs/domains/lakehouse.md
  • docs/domains/dags.md
  • docs/reference/job-layout.md
  • docs/reference/job-execution-modes.md