Freshness And Data-Ready Checks

Summary

Freshness checks make data dependencies visible before automation or review surfaces rely on lakehouse-derived facts.

Problem

Operational jobs often depend on data that was produced earlier by ingestion, silver transforms, gold refreshes, or serving snapshot publication. If a job assumes those products are fresh without checking, failures appear downstream as bad decisions rather than missing prerequisites.

Principle

Treat freshness as a platform dependency. A workflow should know which data products it needs and whether they are ready enough for the action being taken.

Platform Shape

The architecture docs call out freshness and data-ready checks as lakehouse responsibilities. The lakehouse docs describe code-defined DAGs that own scheduled freshness for core incremental data products and residential prospecting products. The DAG docs explain how production DAG definitions and data-product claims live in code.

This lets the platform reason about freshness without scattering ad hoc checks across UI code or individual jobs.

Source Evidence

  • docs/reference/architecture.md
  • docs/domains/lakehouse.md
  • docs/domains/dags.md
  • services/execution/dags/definitions/registry.py
  • docs/engineering/architecture/explicit-publish-steps.md
  • docs/engineering/architecture/lakehouse-service-boundary.md