Using EPIC ZipSync and Pentaho Data Integration to transform and synchronize your data with xMatters

What's data transformation?

Data transformation is the process of extracting, transforming, and loading data from one system into another system. For example, your organization might have an HR system that includes user, group, and device data that you want to load into xMatters for the first time.

Typically, there will be formatting differences between your system's data and the data format that xMatters requires. For example, your phone number format might not match the format used in xMatters, device names might be different, and so on. Depending on the data in your system and how you use xMatters, this process can get pretty complex. Fortunately, ETL software tools such as those offered by Pentaho have been designed to automate and streamline the transformation process.

What's Pentaho?

Pentaho Data Integration is a well-known ETL software tool that we've often used here at xMatters. The attached documentation and files describe how you can use this tool to extract data from your existing system, transform it into the required xMatters format, and then load it into xMatters.

Pentaho is also used by EPIC to drive automated synchronizations to xMatters. The Pentaho jobs and transformations (KJB/KTR files) compare the current HR extract to previous runs to decide which users to add, update, or remove. For example, if a user's "Last Contractual Day of Employment" has passed, or a user disappears from the current extract, the transformation can mark that record with a REMOVE action so that xMatters deletes the corresponding user. If that end date is extended in the source data, Pentaho will continue to produce a PROCESS (keep) action instead, and the user remains active in xMatters.

Does xMatters support Pentaho?

Nope. Don't get us wrong: we're here to assist clients with the data transformation process in general, but we don't have any special relationship with the folks at Pentaho, and we don't provide Pentaho software support (they do, of course).

If you run into issues understanding how a particular field is used to decide whether to remove a user, or why a specific user was or was not deleted during a weekly sync. To diagnose those situations effectively, we'll usually ask you to provide:

• The history of zip files produced by your sync server.

• The user CSV file used by the sync (for example, DS_USERS.csv or User_Input1.csv).

• The Pentaho job and transformation files (KJB and KTR) from your standard sync folder.

• Any sync reports or logs that show which action (for example, PROCESS or REMOVE) was set for the affected user.

Those artifacts let us see exactly what the sync produced and how xMatters applied it.

Are other ETL products available?

Yup. Lots.

How do I get started?

Download the attached documents and files and read through the Data transformation guide for ZipSync. We encourage you to work with the sample files so that you can think through your own data transformation needs - and learn some tips and tricks along the way.

If you're not comfortable configuring Pentaho yourself, contact xMatters Support and we can connect you with our consulting team. They can help design and implement a tailored EPIC/Pentaho data sync for your environment (including handling HR-specific fields, delta logic, and performance tuning), using our newer recommended methods of data synchronization.

xMatters Reference

JDN-4641 Originally created by Cameron Stewart