We had an issue Wednesday with our ServiceNow/xMatters integration where multiple groups were removed and re-added in xMatters while trying to resolve an xmatters role inheritance count issue in ServiceNow.
When the groups were removed and re-added, all shifts/rotations were lost.
The decision was made to revert to our last backup which was from Tuesday. I used Jordan Olin's backup/restore utility to restore just the groups,
https://github.com/xmatters-jolin/xm-labs-capture-instance-data
https://github.com/xmatters-jolin/xm-labs-restore-instance-data
Once that restore was complete, we noticed that the groups and shifts did restore, but the rotations did not. When digging through the backup json file it appears it pulls the rotations, not sure why the restore did not restore this. Also, noticed that approximately 9 groups didn't restore at all, they were all large, complex groups with multiple shifts, and 30+ members. Another group mentioned while the group was restored, it was restored with old data. When the restore file was from the day before. Not sure what happened there. They have all been in contact with me already and are not happy about having to manually recreate their shift configs.
This was my first use of this in Production in a "Disaster Recovery" type event. My testing in Non-prod has been successful, but to be honest our Non-prod environment is only a few groups that do close testing with, and only minimal configurations. It's not like prod, so that's on me.
I'm hoping I don't need to reach out to 270 xMatters groups and ask them to verify and reconfigure their rotations.
Anyone have experience with this python utility? Anyone have any advice?
Comments
Please sign in to leave a comment.
Hi Daniel,
A restore isn't possible for group rotations from an internal perspective. We have the folks that take care of that github integration looking at the matter and we will follow up when they have some idea of whether or not rotations are going to be possible to restore as well via the integration.
Super great question by the way, stay tuned for a further update.
Cheers!
Hey Daniel,
So I have good news and I have bad news. Good news is that it's easy to get the rotations out, it just a one liner change that I'll make shortly.
The bad news is that the APIs don't yet support creating or updating shifts with rotations. I talked to the Product Manager and he is aware and the teams are working to add support for rotations, but no estimated timeframe.
> Also, noticed that approximately 9 groups didn't restore at all, they were all large, complex groups with multiple shifts, and 30+ members. Another group mentioned while the group was restored, it was restored with old data. When the restore file was from the day before.
I don't suppose you had any errors in the log file that might indicate what went wrong here?
Thanks!
--- Travis
I made the change to the capture code to support exporting the rotations, and added an issue to the restore so we can track it.
Please also note to get the code from the official repositories here:
https://github.com/xmatters/xm-labs-capture-instance-data
https://github.com/xmatters/xm-labs-restore-instance-data
Hi Travis,
Could you tell me which capture file was modified? I checked the history link for each file and they all have dates from 2018.
Also, if you want the output log file from the restore, I have it in all it's glory. It's about 3mb is size. It was set at Very, very Verbose :) I don't see a way to attach documents to the communities site.
Please advise.
Thanks - Dan