Dylan's BI Study Notes

My notes about Business Intelligence, Data Warehousing, OLAP, and Master Data Management

Source Dependent Extract and Source Independent Load

Posted by Dylan Wan on March 24, 2015

Typical data warehousing ETL process involves Extract, Transform, and Load.

The concept of Source Dependent Extract (SDE) and Source Independent Load (SIL) are unique part of the BI Apps ETL since BI Apps has a universal data warehouse.

Since the staging schema are designed according to the universal data warehouse design, the logic of loading data from the staging tables to the data warehouse tables are source independent.

Since the staging schema follows the universal data warehouse design, the SDE tasks actually have the specific extraction and mapping logic for each source systems.

Evaluations:

1. Extraction process could impact the performance of the source system since it uses the source system resources.

It typically runs in the limited data warehousing window when the source system does not have much load.

It explains why the change data capture and incremental ETL are important.  We need to use as less resource from source system as possible.

2. The transformation logic could become the overhead.

if the source to target mapping is simply the column mapping, it should not introduce much overhead.  However, if the transformation logic involves heavy calculations, it may become an issue.

It leads the adaption of several technologies:

  • Use Oracle GoldenGate
  • SDS
  • Persistent Staging Tables
  • ODI LKM vs IKM
Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s