Standardise, Prepare and Quality-Assure Data Deliveries with SAP SDI Realtime

Our client from the financial sector wanted to standardise data deliveries to its data warehouse and prepare and quality-assure them in an intermediate HANA system. The goal was to prepare everything for near-time reporting in the data warehouse and ad hoc reporting options for the technical departments in the HANA DB.

Here we show a section of the overall architecture, which has been implemented as part of many other projects. In the following, you can read how we connected approx. 700 tables with a source size of approx. 2 TB from an Oracle system in almost real time and what challenges we had to face. At the time of this excerpt, an MS SQL system had already been built into the architecture via log-based Smart Data Integration (SDI) replication.

© 2021. BIG.Cube GmbH. All rights reserved.

Challenges: Regulating Loading - Recognising Changes

The access was only allowed to take place on the standby node of the Oracle system in order not to load the main node. This standby node was only opened read only. The SAP SDI standard options for real-time require write access to the source system. For the data connection, a possibility had to be found in the SAP context that ensures a fast connection despite the read-only mode of the secondary node. In addition, the subsequent system had to be able to see which data had changed as a result of the loading.

Solution - Smart Delta!

From the connection via SAP SDI to a UI5 app for monitoring the status. We developed a holistic solution. Learn how the challenges were solved in the following steps.

Connection with SAP SDI

SAP SDI, SAP’s own solution for data connection integrated into HANA, was chosen for the connection, which was already used for the connection of an MSSQL system. This made it possible to standardise the loading technology and save on operating costs.

Control Utilisation, Increase Speed

In order to reduce the amount of data that is transmitted and to increase the speed, the delta field in the source system was determined and a delta logic was implemented based on this. This only loads the changed data since the last extraction. To control the load on the agent and the HANA, the tables were divided into classes from S to XL using T-Shirt sizes. By setting the maximum parallel running sizes, the load on the system and the speed of the data update can be controlled depending on the available hardware. The expected frequency of Delta and Delete pick-up can also be set. Here, too, performance can be optimised depending on the infrastructure.

Loading with SDI Flowgraphs

SDI flowgraphs were used for the loading, which are optimally suited for filtering the data and partially complicated delta determination. The last load was passed as input to the flow graph, which then loaded the results into the HANA. The results were enriched with a change date, which can be further used by the data warehouse built on it.

UI5 App for Automatic Data Connection

A SAP UI5 app was developed in order to reduce the operating expenses and the expenses for new tables to be connected. This creates the flow graphs automatically and creates or adapts the corresponding .hdbviews and calculation views. This minimises the effort required to change the data structure. In addition, the effort required for the creation and maintenance of flow graphs, tables and calculation views has been significantly reduced.

UI5 Status Dashboard

To ensure an overview of the status of the data connection at all times, a UI5 app was also created here. This is additionally connected to an automatic ticket creation in Azure DevOps as well as to a mail dispatch. Due to the client’s switch from HANA 1 to HANA 2 with the retention of XS Classic objects and the elimination of the classic admin cockpit, an overview of the replication was urgently needed. Other functions have also been integrated into the cockpit, such as changing the T-shirt size of the tables, setting table priorities, defining table groups or quick access to possible error messages.

All images on this page © 2021. BIG.Cube GmbH. All rights reserved.

Reusability

In our solutions, we always ensure a high level of reusability for our clients. Here, SAP SDI was already set architecturally and could be used for all planned scenarios. During development, attention was already paid to the compatibility of HANA 1 with HANA 2 and the possible use of XSA was prepared. Using SAP SDI, MSSQL, Oracle, S4/HANA and BW on HANA systems were connected to the HANA intermediate layer. The SAP standard for real-time connection was also used alongside the Delta process. This standard can also be controlled via the tools.

Conclusion

With our Smart Delta solution, we were able to standardise our client’s data feeds into their data warehouse in just a few steps. For this purpose, we were able to process and quality-assure the data in an intermediary HANA system – in near-time.

Written by Daniel Fröhler

Share Post

More Exciting Topics from our Newsroom

We are one of “Germany’s best employers”

As one of Germany's top 100 employers with renewed 'Great...

Read More

Shortcuts for SAP BW in Eclipse

This blog post explains five simple shortcuts to make working...

Read More

Bavarian Curling with the BIG.Cube

BIG.Cube employees went bavarian curling together after work. Find out...

Read More