DevOps Playbook

 

Start Here

The DevOps Playbook uses the xMatters communications platform to simulate toolchain integrations. This allows administrators and developers to quickly get a feel for the power of xMatters without spending the time to deploy 3 or more integrations. 

This is one of a series. The others are:

This playbook will simulate an alert from an Application and Performance Monitoring tool (see here for a full list!) about a high CPU usage for the Steelbrick application. The resulting notification will provide several helpful response options ranging from posting the info to a chat room to running a remediation task and rolling back the last commit. 

 

Setup

Navigate to the Developer tab and click the Import Plan button. 

 

Navigate to the Integration Builder and expand the Inbound Integrations:

 

... and click on "Inbound for Conference Bridge", scroll down and copy the URL. 

 

 

Click the breadcrumbs to go back to the Integration Builder and click Edit Constants. Select the "Playbook Status Update Endpoint" and paste in the path (or the full url, either will work) and click Save. Repeat for "Inbound for Post to Chat" and "Post to Chat Endpoint" and "Inbound for Rollback Last Commit" and "Rollback Last Commit Endpoint". 

 

 

Following the Playbook

Think of the playbook as a Choose Your Own Adventure, where you start at point A, but then get a choice to go to point B, C, or D. Or even all of the above! So let's start adventuring. 

Navigate to the Messaging tab and in the DevOps Playbook section, click APM Alert. Default information has been entered for you, except for the recipient. Enter your username into the recipient box and click on the user entry. Once you are satisfied, click the Send Message button. 

 

All enabled devices will be targeted with the notification content appropriate for that device. 

Mobile/Email:

SMS:

 

 

The next sections go over what happens with each response option. 

Assign to Me

This will simulate reaching back into the ITSM application to set the responder, you again, as the assignee. In a real world environment, this is considered a "closed loop" integration because the event is generated in the ITSM application, fired to xMatters, people are notified, they respond and that information is delivered back to the application. This helps keep your teams informed and confident that someone has owned the incident. 

Create Chat Room

This response option simulates creating a chat room in your favorite chat application with the room name matching the major incident number. Of course any chat room could be used, but matching the chat room with the incident number can help in any root cause analysis that needs to be done. From here, you could use the integrations we have to these chat applications to further invite others to assist in quickly getting the major incident resolved. 

Start Conference Bridge

Does a voice to voice collaboration fit the style of your team? Or do upper management types like to drop in to get updates and don't bother to read email? This is a good one for you then. This will actually generate an xMatters hosted conference bridge and invite the responder, guess who? you!, to the bridge with a follow up phone call. For this particular playbook we aren't inviting anyone else so you will just hear music.

The form that launches this bridge is enabled for the Web and Mobile UI, so if you'd like to try it out with your teammates, check out the section below

Inform Stakeholders

Would your upper management types or other interested parties rather just get passive updates, this is the go-to response option. When selecting this from a mobile device or email, make sure to enter some comments as entering the comments is what kicks off the event with the updates you enter. These comments, turned update, will be included in the outgoing notification, in this case, to the responder, but in a real life scenario can be targeted to a group or users. Alternatively, a subscription panel could be created to allow for users to decide what criteria they'd like to be updated on. 

Resolve

Similar to the Assign to Me response option, in a real-life scenario, this will reach into the ITSM application and resolve the issue. For this playbook, the response is just noted. 

Escalate

The Escalate response option is used to tell xMatters that you aren't available to deal with this, but instead of waiting for you to reach out to someone, this will immediately escalate to the next on-call person. In this playbook scenario, since we only targeted one person, the response is simply noted and no further action is taken.  

Alternative Adventures

In the Messaging tab, under the "MIM Playbook" section, there are two other message panels. Details on these are provided below.

Major Incident

This message panel is great for simulating a major incident from your favorite ITSM application. The data is pre-populated, but you are free to edit to your heart's content. Enter some recipients and hit Send Message!

 

 

 

 Conference Bridge

The Conference Bridge form allows you to invite other teammates to a conference bridge. The "Bridge" drop down will have entries for any in-progress xMatters conference bridges so you can quickly invite other members to collaborate. 

 

Troubleshooting tips

I didn't receive any notifications

Uh oh! That's not good. First, check the Reports tab to make sure the event was created. If the event was created, then you'll see a nice happy entry in there, like so:

Clicking on the link will display a dashboard breaking down the recipients:

.. and clicking on each one will show more information. Make sure there are no errors and that the Delivered shows at least 1. 

 

I didn't receive an email/phone call/SMS/push notification

Ok, so you got the notification on some devices, but not all the ones you expected. First, check out your devices by clicking the profile icon in the upper right and select Devices. 

Well, here's a problem, I have a 10004 minute delay after my email! Yea, that's not going to work:

 

If the delays are in good order, click on the Options drop down for each and make sure the device is enabled and the schedule is as expected:

 

You can also see pertinent information in the Event Tracking Report on the Reports tab. This will show any system errors encountered delivering to the device. 

 

 

Choosing a response didn't send me another notification

Well, so not ALL of the responses will send a notification. The Escalate and End response options won't trigger an additional notification.

Outside of those, you should have received a new notification. So we might need to do a little more digging. First, review the Setup section and make sure the constants are set properly. If those look ok, we'll have to dig even deeper. 

Head over to the Developer tab and click Edit > Integration Builder next to the MIM Playbook Comm Plan.

 

Then, expand the Outbound integrations and click the gear icon next to the response option you chose and click the Activity Stream.

If it is blank and logging is turned off, then flip the slider and try your response option again, otherwise inspect for any errors. 

If you're still not sure, reply in the comments below or open up a ticket with our cheerful support peeps. 

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk