0

xMatters Restore

We had an issue Wednesday with our ServiceNow/xMatters integration where multiple groups were removed and re-added in xMatters while trying to resolve an xmatters role inheritance count issue in ServiceNow.

When the groups were removed and re-added, all shifts/rotations were lost.

The decision was made to revert to our last backup which was from Tuesday. I used Jordan Olin's backup/restore utility to restore just the groups,

https://github.com/xmatters-jolin/xm-labs-capture-instance-data

https://github.com/xmatters-jolin/xm-labs-restore-instance-data

Once that restore was complete, we noticed that the groups and shifts did restore, but the rotations did not. When digging through the backup json file it appears it pulls the rotations, not sure why the restore did not restore this. Also, noticed that approximately 9 groups didn't restore at all, they were all large, complex groups with multiple shifts, and 30+ members.  Another group mentioned while the group was restored, it was restored with old data. When the restore file was from the day before. Not sure what happened there. They have all been in contact with me already and are not happy about having to manually recreate their shift configs.

This was my first use of this in Production in a "Disaster Recovery" type event. My testing in Non-prod has been successful, but to be honest our Non-prod environment is only a few groups that do close testing with, and only minimal configurations. It's not like prod, so that's on me. 

I'm hoping I don't need to reach out to 270 xMatters groups and ask them to verify and reconfigure their rotations.

Anyone have experience with this python utility? Anyone have any advice?

4 comments

Please sign in to leave a comment.