Source Dependent Extract and Source Independent Load
Posted by Dylan Wan on March 24, 2015
Typical data warehousing ETL process involves Extract, Transform, and Load.
The concept of Source Dependent Extract (SDE) and Source Independent Load (SIL) are unique part of the BI Apps ETL since BI Apps has a universal data warehouse.
Since the staging schema are designed according to the universal data warehouse design, the logic of loading data from the staging tables to the data warehouse tables are source independent.
Since the staging schema follows the universal data warehouse design, the SDE tasks actually have the specific extraction and mapping logic for each source systems.
1. Extraction process could impact the performance of the source system since it uses the source system resources.
It typically runs in the limited data warehousing window when the source system does not have much load.
It explains why the change data capture and incremental ETL are important. We need to use as less resource from source system as possible.
2. The transformation logic could become the overhead.
if the source to target mapping is simply the column mapping, it should not introduce much overhead. However, if the transformation logic involves heavy calculations, it may become an issue.
It leads the adaption of several technologies:
- Use Oracle GoldenGate
- Persistent Staging Tables
- ODI LKM vs IKM