Posted by Dylan Wan on October 30, 2015
Almost all data warehouse have a date dimension. The purpose of the date dimension is to provide some pre-calculated grouping for dates. It helps rolling up the data that entered against dates to a higher level, such as year, quarter, month, week, etc.
In some system, source files are used in generating the date dimension. IMHO, it makes the modification to the logic difficult. In some ETL programs, the task involves various table joins, try to generate the rows for the year range.
This post is for describing how to populate a table with rows for each date for a given year range. Read the rest of this entry »
Posted in Common Dimension, Data Warehouse | Tagged: Data Warehouse | Leave a Comment »
Posted by Dylan Wan on October 4, 2015
These are different concepts.
Data Lake – Collect data from various sources in a central place. The data are stored in the original form. Big data technologies are used and thus the typical data storage is Hadoop HDFS.
Data Warehouse – “Traditional” way of collecting data from various sources for reporting. The data are consolidated and are integrated. A data warehouse design that follow the dimensional modeling technique may store data in star schema with fact tables and dimension tables. Typically a relational database is used.
If we look at the Analytics platform at Ebay from this linkedin slideshare and this 2013 article: Read the rest of this entry »
Posted in Big Data, Data Warehouse, EDW | Tagged: big-data, Data Warehouse, hadoop | Leave a Comment »
Posted by Dylan Wan on September 22, 2015
I feel that these are the rules applicable for any cloud based data warehouse solution. In general, I feel that the on-premise data warehouse deployment probably will remain for a while.
1. For a columnar database, “select *” is bad
I think that the projection needs to be done as early as possible and should be pushed down.
If a column is not needed in the downstream flow, it should not be selected in the first place.
If the application logic is defined in the metadata, the tool should read it and generate the extract logic. Read the rest of this entry »
Posted in Business Intelligence | Tagged: BI APps, BI Cloud Service, BICS, Cloud BI, Data Warehouse | Leave a Comment »
Posted by Dylan Wan on March 23, 2015
BI Apps data warehouse design is based on an assumption that the data warehouse schema design is independent from OLTP system.
The staging schema is an universal staging and the data warehouse is an universal data warehouse.
The assumption is that no matter what the source system you are using, the business questions the BI consumers have are similar. You want to see revenue, cost, and profitability, regardless if you are using Oracle E-Business Suite, PeopleSoft, or J.D. Edwards, revenue are still revenue, cost are still cost. The definitions of these metrics won’t vary too much from the OLTP systems you are using.
It is the BI Apps development team’s job to find where this numbers should be extracted from the source systems and to define the “map” between the source system and the target data warehouse system.
This universal data warehouse assumption does give BI Apps a unique advantage. It allows the teams develop the source adaptor independently from the data warehouse design and thus independently from the BI metadata.
In another way to view this, the application logics built in the BI tool and the knowledge of designing the data warehouse schema can be reused over time and over many projects.
Adding the support of the 1st adaptor to E-Business Suite may take sometime to analyze the analytic requirements. Adding the support of the PeopleSoft and JDE adaptors becomes a job of providing the source and target logical mapping and create the ETL jobs of loading the universal staging tables.
Posted in Business Intelligence | Tagged: BI, BIAPPS, Data Warehouse | 1 Comment »
Posted by Dylan Wan on October 6, 2014
Prepackaged analytics applications are available as cloud services.
The idea is that the client company does not need to use their own hardware and does not need to install the software or apply patches by themselves.
What they need is just simply the browsers.
For the end users, there should not be much difference. The BI apps built on the OBIEE platform is already a web based application. Users today, even when they use the BI applications deployed on premise, access the applications via the browser.
The real difference is to the IT department and to the company from the investment perspectives.
The deployment time should be very short.
Where is the role of the ISV, VAR, or SI in this picture?
I think that their role is still very important.
First of all, the out of box reports may not work for everyone. New reports may need to be created.
Secondly, the source apps may be extended. For example, oracle cloud services, such as CX clouds, support extensible attributes. Where those attribute need to be reported should be probably added or hidden.
When the data are not loaded from other Oracle cloud applications, loading the data from Oracle’s Application Unlimited products such as PeopleSoft, E-Business Suite, Siebel, J.D. Edwards may require a lot of work.
Posted in Business Intelligence | Tagged: Cloud BI, Data Warehouse, Extensibility, OTBI-E | Leave a Comment »