Incremental Source Seeding

Summary

Incremental source seeding keeps bronze mirrors current without replaying every source system from scratch on every run.

Problem

External systems are large, rate-limited, and operationally active. Full reloads are useful for backfills, but normal platform freshness needs narrower windows that can run safely and repeatedly.

Principle

Make ingestion parameterized and repeatable. Jobs should expose explicit modes, while pipelines own the deterministic source reads and persistence behavior.

Platform Shape

The lakehouse docs describe incremental seeds for ServiceTitan, QuickBooks Online, and Spectrum. The executable job wrappers under jobs/lakehouse/ run those pipeline sequences, while ingestion code lives under pipelines/lakehouse/ingest.

Source Evidence

• docs/domains/lakehouse.md
• pipelines/lakehouse/ingest
• jobs/lakehouse/servicetitan_incremental_seed/run.py

Related Reading

• docs/engineering/concepts/bronze-ingestion.md
• docs/engineering/reference/job-entrypoint-contract.md