Extracting Data from Cloud Apps
Posted by Dylan Wan on March 5, 2015
I think that it would be easier if the cloud application can be aware of the data integration needs and publish the interfaces proactively.
Here are some basic requirements for the applications that can be considered as data integration friendly:
1. Publish the object data model
This is required for source analysis.
For example, here is the RightNow Object Model.
The documentation should describe the object, the attributes available, the identifiers, the validation rules, the relationships among the objects, including their cardinality, the domain or lookup used in validating the data, etc.
2. Support a scalable solution for the Full Extract
Initial load may require a full extract of the historical transaction data and reference data. This is typically a one-time only event during implementation phase.
If the extract starts about the same time when the cloud apps is initially implemented, the volume of the initial extract may be similar to incremental extract. However, if the Cloud BI is implemented much later, after the transational apps, the volume of initial load may be big.
3. Support incremental extract / Change Data Capture
The daily or near real-time extract requires a way to identify the delta, the data created or updated recently.
In many systems, this can be done via an auditing column, such as the last updated date column. However, if the source object model allows a nested structure. The last update date may need to be available in two forms, update the header only, or update any objects including the child and the descendant.
4. Delete Capture
Capturing the deleted data during the incremental extract is different from the creation and update cases. If the deleted rows are not extracted, the data will be kept in the data warehouse and be treated as a normal data. The report could be considered wrong.
In some source systems, deletion and archiving are different events. We need to have a way to differentiate these two events.
5. Metadata Extract
This is a more advanced feature. It is required for supporting the custom objects.
In other normal objects, the expected data model is determined during the design time.
However, if the source system allows customization that can add or enable new attributes or even creating new objects, the extraction program needs to be able to detect the model change and react accordingly.
6. Extracting the Lookup or Domain data
Many systems support the multi-language using lookup table. The idea is that the translated strings are kept in a separate table and allow the data shown on the application UI in different languages depending on the preferred language from the login user. Not all source system can support this feature. However, for the source systems that can support this feature, extracting the lookup data can allow the BI showing the data in the same translated as the source system.
The requirement for the extracting process is to extract the data in all languages that are expected in the BI / Data Warehouse side.
7. Extracting data for integrated data security
If the users that can access the cloud source apps and the users that can access BI are the same set of users, or a subset, the user and authentication service should be shared.
However, if the user or the internal person record identifiers are used in building the data security, extracting the users and the roles granted may be necessary.
8. Extracting the UI displayed Labels
In some cases, the cloud BI can be a seamless integrated or even embedded into the source apps. The displayed lable for the object and for the attributes need to be consistent. In such cases, extracting the display labels are also requried.
It may be similar to the process of extracting the lookup or domain data which have the data available in multiple languages.
This entry was posted on March 5, 2015 at 1:20 pm and is filed under Business Intelligence. Tagged: BI, BIAPPS, Cloud BI, Data Warehouse, Embedded BI, Extensibility. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.