Major Incident Management Playbook

 

Start Here

This Playbook uses the xMatters communications platform to simulate toolchain integrations. This lets you use a sample integration to quickly get a feel for how xMatters works without a lot of manual set up work.

Playbook Scenario

ACME Corp. has xMatters integrated into their system. Their monitoring tool (see here for a full list of our monitoring integrations!) sends an alert, indicating a slow user response rate in the ACME App. Let's see how this plays out with xMatters.

MIM-example.png

Setting up the playbook

In the playbook, we use constants in xMatters to simulate parts of the process, such as the push to the ITSM application or starting a conference bridge, so there's a bit of setup that you have to do.

You need access to an xMatters instance, of course, and a device on which you can receive notifications. (If you want to try the conference call scenario, make sure you have a voice device handy...)

Download the sample integration:

To use this playbook, first you need to download the sample integration. (Don't worry, clicking the link just takes you to the bottom of the page.)

Now that you've got the file downloaded, let's import the playbook and get it set up.

Get the playbook ready to run:

  1. Navigate to the Developer tab and click Import Plan.
    • If you get warnings about languages, it just means you have languages enabled in your instance that we don't have translations for in the sample plan.

ImportPlan.png

  1. After the import is done, click Edit beside the plan and select Integration Builder, then expand the list of inbound integrations:

IntegrationBuilderMIM.png

  1. Click "Inbound for Conference Bridge", scroll down, and copy the URL.

CopyURL.png

  1. In the real-world, you'd copy this URL into a webhook in your APM. But for our simulation, we need to use constants to mimic that webhook. So...click the breadcrumbs to go back to the Integration Builder and click Edit Constants.
  2. Select the "Conference Bridge Endpoint" and paste in the path (or the full url, either works) in the Value field and click Save Changes.

MIM_constants.png

  1. Repeat for "Inbound for Create Chat" > "Create Chat Endpoint", "Inbound for Inform Stakeholders" > "Inform Stakeholders Endpoint", and "Inbound for Major Incident" > "Major Incident Endpoint".

That's it – you're ready to play.

Following the Playbook

Think of the playbook as an ever-changing adventure, where you start at point A, but then get a choice to go to point B, C, or D. Or even all of the above! So let's start adventuring.

Navigate to the Messaging tab and, in the Major Incident Management Playbook section, click Monitoring Alert. Default information has been entered for you, except for the recipient – that's you! So, in the recipient field, start typing your username to set yourself as the recipient.

Once you're satisfied, click Send Message.

MIM_MonitoringAlert.png

All enabled devices are targeted with the notification content appropriate for that device.

Mobile/Email:

MIM_MonitoringAlertApp.png

SMS:

MIM_MonitoringAlertSMS.png

For this adventure, choose "Create Major Incident". This response simulates creating a new Major Incident in your favorite ITSM application, and generates a new event in xMatters targeting the appropriate responder (yup, you again).

MIM_MajorIncidentAlert.png

This new notification now has a different set of response options, each of which we'll discuss individually.

Assign to Me

This response simulates reaching back into the ITSM application to set the responder, you again, as the assignee. In a real-world environment, this is considered a "closed-loop" integration because the event is generated in the ITSM application, fired to xMatters, people are notified, they respond and that information is delivered back to the application. This helps keep your teams informed and confident that someone has owned the incident.

Create Chat Room

This response option simulates creating a chat room in your favorite chat application with the room name matching the major incident number. Of course, any chat room can be used, but matching the chat room with the incident number can help in any root cause analysis that needs to be done. From here, you could use the integrations we have to these chat applications to further invite others to assist in quickly getting the major incident resolved.

Start Conference Bridge

Does a voice-to-voice collaboration fit the style of your team? Or do upper management types like to drop in to get updates and don't bother to read email? This is a good one for you then. This actually generates an xMatters hosted conference bridge and invites the responder - guess who? you! - to the bridge with a phone call. For this particular playbook, we aren't inviting anyone else so you'll just hear music.

The form that launches this bridge is enabled for the Web and Mobile UI, so if you'd like to try it out with your teammates, check out the section below.

Inform Stakeholders

This is the go-to response option if your upper management types or other interested parties would rather just get passive updates. When selecting this from a mobile device or email, make sure to enter some comments since it's the comments that kick off the Inform Stakeholders notification.

The selected response has been sent to xMatters, but to trigger the notification to the stakeholders, you still need to add a comment:

MIM_InformCommentsDialog.png

Here is another look on mobile:

MIM_InformCommentsApp.png

These comments are sent as updates to the stakeholders (in this case, you with your VIP hat on but in a real-life scenario, it can target whatever groups or users you want). Another way to keep stakeholders informed is using subscriptions, which allow users to decide what criteria they'd like to be updated on.

Resolve

Similar to the Assign to Me response option, in a real-life scenario, this reaches into the ITSM application and resolves the issue. For this playbook, the response is just noted.

Escalate

Are you busy responding to another issue? Or down with the flu, holed up under a blanket, and didn't set a replacement? The Escalate response option lets you tell xMatters that you aren't available to deal with the issue, so xMatters can immediately push the alert to the next on-call person. In this playbook scenario, since we only targeted one person, the response is simply noted and no further action is taken.

Alternative routes

In the Messaging tab, under the "MIM Playbook" section, there are two other message panels you can try out.

Major Incident

This message panel simulates a major incident from your favorite ITSM application. It basically functions the same as the "Create Major Incident" response, without being triggered by an alert from your monitoring application. The data is pre-populated, but feel free to edit to your heart's content. Enter some recipients and hit Send Message!

MIM_MajorIncidentForm.png

Conference Bridge

The Conference Bridge form allows you to invite other teammates to a conference bridge. This is also the form triggered when you select the Start Conference Bridge response on the Major Incident alert.

The "Bridge" dropdown has entries for any in-progress xMatters conference bridges so you can quickly invite other members to collaborate.

MIM_ConferenceForm.png

Ready for your next adventure?

Try out one of our other playbooks:

If you want to dive right in, set up one of our monitoring integrations, then link it to one of our ITSM or chat integrations, as demonstrated in the playbook.

Or try something new - connect it to our Statuspage integration: When the monitoring application alerts xMatters of an issue, add a response option that updates your Statuspage to inform users you're aware of the issue and working on it.

Troubleshooting tips

I didn't receive any notifications

Uh oh! That's not good. First, check the Reports tab to make sure the event was created. If the event was created, then you'll see a nice happy entry in there, like so:

MIM_reports.png

You can click on the title to display a dashboard breaking down the delivery:

MIM_trackingReport.png

... and clicking on each one will show more information. Make sure there are no errors and that the Delivered shows at least 1.

I didn't receive an email/phone call/SMS/push notification

Okay, so you got the notification on some devices, but not all the ones you expected. First, check out your devices by clicking the profile icon in the upper right and select Devices.

ProfileDevices.png

Well, here's a problem, I have a 10004 minute delay after my email! Yea, that's not going to work:

DeviceSetup.png

If the delays are in good order, click on the Options drop down for each and make sure the device is enabled and the schedule is as expected:

DeviceOptions.png

You can also see pertinent information in the Tracking Report on the Reports tab. This shows any system errors encountered delivering to the device. You can also check out the Log tab in the event report.

EventLog.png

Choosing a response didn't send me another notification

Well, so not ALL of the responses will send a notification. The Accept, Escalate and End responses in the Monitoring Alert and the Assign to Me, Resolve and Escalate responses in the Create Major Incident alert won't trigger an additional notification. Also, the Inform Stakeholders response only fires a new message if you enter comments in the dialog box or the text field in the mobile app.

Outside of those, you should have received a new notification. So we might need to do a little more digging. First, review the Setup section and make sure the constants are set properly. If those look ok, we'll have to dig a bit deeper.

Head over to the Developer tab and click Edit > Integration Builder next to the Major Incident Management Playbook Communication Plan.

EditIB.png

Then, expand the Outbound integrations and click the gear icon next to the response option you chose and click the Activity Stream.

IntegrationActivityStream.png

If it is blank, logging might be turned off. Flip the slider and try your response option again; if that doesn't work, inspect the Activity Stream for any errors.

If you're still not sure, reply in the comments below or open up a ticket with our cheerful support folks.

Download the playbook

Attachments
Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.