Data Boundaries

Summary

LKCI currently uses one Postgres database for several kinds of truth. That is practical, but it is also dangerous unless the code treats ownership as stricter than storage. Bronze, silver, gold, serving projections, OLTP workflow state, and ops execution state can live near each other physically while still having different rules, writers, and consumers.

The boundary is not a schema naming preference. It is how the platform remains split-able later and understandable now. A reader should be able to ask of any fact: where did it come from, who owns it, who may write it, when is it safe to consume, and what kind of repair changes it?

Module At A Glance

The Reader's First Question

Most boundary violations start from a reasonable UI or workflow question. Why is this customer visible? Why did a campaign audience include this prospect? Why is a payroll row stale? Why did a morning brief omit a fact? Why does a geo layer show this score? The fast answer is often to query the table that appears to contain the data. The correct answer is to identify which layer owns the fact and which projection the surface is allowed to use.

That distinction matters because the same business object can appear in multiple forms. A provider's raw customer payload belongs in bronze. A canonical customer identity belongs in silver. A campaign response or membership summary may belong in gold. A compact dashboard projection may belong in serving. A human repair decision belongs in OLTP state. A job run result belongs in ops. They can all refer to the same real-world entity while answering different questions.

Publication Lifecycle

The lakehouse lifecycle is source capture, bronze preservation, silver canonicalization, gold curation, publish, serving, and freshness validation. Bronze keeps source shape so ingestion can be replayed and audited. Silver normalizes identities, locations, jobs, events, crosswalks, and other canonical entities. Gold turns canonical data into analytical or operational facts. Serving snapshots and read models give UI/API surfaces stable shapes to consume.

Publishing is the hinge. A pipeline can rebuild an internal fact many times while source logic improves. A UI surface should not observe every intermediate state. A publish step says that a slice is ready for downstream use. A serving snapshot says that this is the contract a consumer may read without knowing pipeline internals.

Freshness and data-ready checks belong at the same boundary. A marketing run can ask whether an audience source is ready. A geo console can show whether a score vintage is current. A morning brief can validate required facts. A payroll review can refuse to proceed if source rows are stale. Those checks should be services or published signals, not one-off component logic.

OLTP Is Operational Memory

OLTP state is different because it records active work. Cases, issues, reviews, approvals, repairs, run members, delivery attempts, action executions, operator decisions, and provider command outcomes should not disappear when a lakehouse product is rebuilt. They are operational memory.

This is the point that makes the boundary more than a data-engineering concern. If an operator corrects a POA interpretation, that correction is not a new bronze fact. If a marketing asset is approved, that approval is not a gold rollup. If a payroll sync is blocked, the blocker is not a source-system row. If a notification provider rejects a send, the delivery outcome is not just an email analytics event. These records explain what the company did.

Operational state may reference lakehouse-derived identifiers, but it should not depend on direct reconstruction from raw analytics tables. That is what lets workflows remain auditable when source data changes.

Concrete Boundary Examples

Marketing is a strong example. Audience source facts may come from lakehouse products or geo selections, but a campaign run materializes its own members, member steps, rendered assets, deliveries, events, and action plans. The run must remain explainable even after an audience definition or template changes. Analytics jobs can later publish lifecycle and attribution facts, but they do not own the run's review state.

Geo is different. The geo schema owns parcel features, H3 score rows, road network artifacts, and saved selections. These are analytical and domain-owned spatial products. Marketing may import a saved selection, but it should not query geo tables directly to create campaign members. The saved selection is the boundary artifact.

Morning brief shows the publication side. Gold context and payload products prepare reusable facts. Email generation reads the published payload rather than assembling final communication directly from bronze provider tables. The notification layer then owns delivery state and provider diagnostics.

Boundary Smells

A UI component joining raw lakehouse tables is a smell. A pipeline mutating review state directly is a smell. A job that writes provider outcomes into analytical history without preserving command state is a smell. A route that combines request parsing, SQL joins, business policy, and provider calls is a smell. A service that accepts unstructured dictionaries for core data flow is making future consumers guess at contracts.

The fix is usually smaller than the smell suggests. Publish a serving projection. Add a typed contract. Move a mutation behind a service. Create a review issue instead of swallowing ambiguity. Record provider outcome as command or delivery state. Add a freshness check at the publish boundary. The goal is not ceremony; the goal is to make ownership visible.

Designing With The Boundary

When designing a new feature, decide the owner of each fact before deciding where to query it. If the fact is raw source history, keep it in ingestion. If it is canonical identity, put it behind the lakehouse service or silver/gold contract. If it is a curated analytical result, publish it. If it is an operator decision, store it in OLTP state with tenant, actor, timestamp, and audit context. If it is an external side effect, record command, attempt, idempotency, provider result, and retry state.

This discipline is what makes a future physical split possible. If UI code, jobs, and domain services already respect conceptual ownership, moving OLTP and analytics into separate stores becomes an infrastructure change with targeted adapters. If everything freely joins everything else, the split becomes a platform rewrite.

What Changes A Fact

The cleanest boundary test is to ask what kind of event is allowed to change a fact. A source refresh can change bronze and downstream canonical facts. A pipeline correction can change silver or gold outputs. A publish step can change what a consumer sees. An operator approval can change OLTP workflow state. A provider retry can change command or delivery state. These are different causes, and they should leave different evidence.

That distinction is useful during incidents. If a dashboard changed because a gold product republished, the repair path is data lineage and freshness. If a campaign member is blocked because a review decision is missing, the repair path is workflow state. If a message failed because a provider rejected it, the repair path is delivery diagnostics. Treating all three as "data issues" makes support slower and architecture blurrier.

Tests and hooks should protect the boundary where it becomes precise. A route guard can prevent direct analytics access. A service test can prove a publish contract. A job test can prove execution remains a thin wrapper. A domain test can prove a repair action changes OLTP state rather than analytical history.

The point is not to prevent engineers from using data. It is to make data safe to use repeatedly. A fact that has a clear owner, publish boundary, freshness signal, and repair path can be reused by many workflows without every consumer relearning its lineage. That is what turns data from a local dependency into a platform capability.

Visual

The publication state-machine diagram should be read as the data contract for operational consumers: source capture, bronze, silver, gold, quality checks, publish, serving, stale, and failed states. The important part is the handoff. Downstream workflows should consume published or serving-ready facts, not pipeline internals.

Source Evidence

  • docs/reference/architecture.md
  • docs/domains/lakehouse.md
  • services/lkci_lakehouse
  • pipelines/lakehouse