Predictive Analytics and AI/ML

We have heard about advanced analytics, which was described by dividing analytics into three main types:

• Descriptive
• Predictive
• Prescriptive

While many articles explore these concepts,, such as Prescriptive vs. Predictive Analytics: Examples & Use Cases

I aim to approach these categories from the standpoint of a software vendor, describe how these categories relate to machine learning practices.

Migrating from OBI to Incorta

I am sharing my experience of migrating from OBI to Incorta.

Process

1. Start with Incorta EBS Blueprint
2. Configure and customize for the deploying company
3. Optionally, Demo the Fusion Connector
4. Preview and demo to business users using their own data
5. Provide the existing OBI dashboard usage analysis – Help prioritize the replacement project
6. Provide the lineage and SQLs from OBI. Get the access to nqquery log and SDE logic to the development team
7. Analyze the SQL and OBI report one by one against the blueprint business schemas
8. Interactive processes about enhancing Incorta physical schema and creating dashboards in Incorta
9. Create materialized View when necessary
10. Demo and get feedback from business users during the development
11. Leverage the existing OBI team in verifying and comparison, help the team gain knowledge about Incorta for future support
12. Prepare the pilot and overlap period before shutting down OBI. Define the Exit criteria
13. Prepare the training to the future analyzers/superuser, who can create and manage their own contents. This may be necessary as a recursive event for several months. Providing an office hour will be helpful.
14. Provide a demo to Excel Add-in if necessary
15. Provide the SQL interface if necessary. Control the usage by understanding their usage. This may be helpful for Tableau users, but need to be careful as it impacts the system resource usage
16. Schedule the dashboard delivery and the download. Configure the integration with network drive storage. FTP, dropbox, OneDrive, GDrive, etc.
17. Along with the development of the new content, start discussion about job schedule options. For example, dependencies from Incorta to source app batch process. Also, understand the source data update pattern and frequency. For example, when the Transfer to GL was executed, when the depreciation process is executed. How frequent these processes are run. For example, any difference in period closing weeks?

Very few resistance from existing OBI developers since most of them see this as an opportunity.

Learning new skills such as using Spark SQL and PySpark in creating materialized views. Explore ML library in Incorta.

Their Data Model knowledge is typically the asset to the new platform.

Oracle App Cloud and Incorta

OTBI is great. But when people are migrating from Oracle EBS to Oracle Cloud App, they would like to view the data from both EBS and Oracle Cloud, Incorta becomes a cost saving and a quick implementation solution without implementing a data warehouse.

Incorta is not a data warehouse although it does has the data storage, which is optimized for extraction, optimized for internal process reading and transformation, and optimized for analytical queries.

The scalability is accomplished by a distributed architecture, and thus no issue with resource contention.

The support of Tableau and PowerBI provides an alternatives for those companies that already standardized their BI front-end.

Very interested to see how Incorta can help upsell Oracle Cloud App, not the other way around.

Scalable Distributed BI Architecture

I have not written anything for almost a year.  I think that one of the major change is that I am dealing with a distributed BI architecture.

There is not much extra work for a solution architect to do in a distributed BI architecture.  Design of schema and dashboards are the same.  The metadata are the same.  The challenge is all on the system, but supporting it including debugging the issues is new. Understanding how it works itself is interesting.

High Availability – The BI system needs to be always available.  If a machine is down, can the site be still up?  For the front-end UI, we need to have a load balancer that is the site accessed by users.  We can have multiple nodes that serve requests from the users, but which node is being used is random, or can be configured based on rules.  Having HA means that the nodes involved are providing the same services so they can replace each other.  Even one node failed, the other node can still take care of new requests.

Disaster Recovery – A stable BI system needs to support recovery.  DR is about resuming the operation from a disaster.  It is different from HA in that the nodes in HA are serving requests.  DR site is a backup and may be passive, not active. DR site typically has some distance from the main site so it won’t experience the same disaster.

Horizontal scaling – Achieving horizontal scaling may also be involved in multiple distributed nodes but it is a bit different.  This is about reacting to the growth of your organization and thus the growth of the data. You can add more nodes to serve new request is horizontal scaling, but having nodes that are providing the same servicing without distinguishing the existing and new can be more desirable for achieving horizontal scaling.

Separation of Data Preparation/Integration and Data Request – Many conventional BI and ETL systems are separate by nature.  But breaking a system that has been originally designed to be one JVM and one process to be separate but integrated systems is equally challenge.  Data consistency requires that the data that are updated not available unless other integrated parts are also available.  Latency is OK but inconsistent is not OK.  While the data is being updated, the users who rely on the data to make decision should not be affected, until the last moment that the new consistent data becomes available.

Technologies

Zookeeper –   I don’t want to use the term database to describe it but it is indeed a database that store a specific type of data – the messages between nodes.

Apache Helix – A cluster management A good slideshare.

Shared Metadata Database

Shared File System

An enterprise level BI architecture is not just provide instant queries and not just about providing the minimal latency.  It is about highly available, supporting disaster recovery, and being scalable over time.  Different types of challenges indicate how the deploying company is relying on the system and reflects the stage of the product and company in its lifecycle.

 

 

Users of Analytics Applications

Business User who are consuming the data and the report.  They see the information pushed to them.  They can see alerts in their phone.  They see emails.  They add the page to a bookmark in their browser and periodically look at them.   They are executives, managers, busy users who have other duties.   They don’t spend much time.  They may not come to see the data every day.  In a school system, they are teachers, principals, vice principals, school counselors.  In an corporation, they are the business partners of IT.  They are controllers, account payables, contractor admin, buyers, billing clerks.  Many of them know what information they would like to have and information is essential for their jobs.  They do not like complex navigation.  They would like to see dashboards or data integrated into their portal or the applications they have to use.

HR Reporting

HR is an important business function in any company.  It supports business growth.  It also satisfies the compliance requirements.  Reports are required for supporting the HR functions.

Typical HR Reporting Requirements

HR reports are surrounding the HR function:  Human Resource Planning, Recruitment and Selection, Orientation and Training, Performance Appraisal, Compensation and Benefit.

They serves many different roles and thus having different ways to slice and dice the data.

• Personal Information: Demographics, Education, Job, Location, Manager, Shift, Salaries, Payroll, Skills, Benefit
• Personal Information Changes – Before and After for audit and review
• New Hire – Who are they?  Where we are in the hiring status?  When they will start?  Temporary or Employment?  Full Time or Part Time, New Hire or Rehire? Have they attended the orientation?   Are they getting sign-on Bonus?  Are they getting stock options?
• Allowance – Who got the allowance?  How much?  How it was calculated?
• Transfer – internal transfers, what is the releasing org? which org the person is moving to?
• Assignment – Global, special assignments, start date, end date, purpose, etc.
• Training – Who take training?  What training? Compliance Status
• Termination – Who have left?  Who is leaving?  When?  Last Payroll date?

Challenges

I found that HR reporting is somewhat challenging: