1

Discussion Topic: Common Blockers to Incident Management

Mario Guisado

March 11, 2020 20:06

None

Follow

Hello xMatters Community,

Here at xMatters Customer Support there's been a LOT of activity around incident management from customers recently, which makes sense with the current climate and its impact to business.

We created this forum to help foster discussion around incident management in general, and so you can share your ideas and use cases, or just ask what everyone else is doing. We'll share ours too.

The first step is to develop a plan, though in my experience it isn't as easy as it seems. Here are some of the blockers that I've seen throughout my career. What are yours? How do we address these so that we can move forward?

• Facebook
• Twitter
• LinkedIn

2 comments

• 1
  
  Mark Henzi March 23, 2020 22:00

  Limited Practice was an interesting problem for us in our engineering teams.  The challenge was 2-fold:

  1. As part of the move to the cloud, many teams were tasked with becoming front-line responders for the first time.  They knew the general workings of an incident (I mean, it is kind of core to our business after all) but few had ever really "worked" an incident.

  2. Our core incident response teams were literally out of practice after a long period of stability.  We had the occasional incident, but our monitoring was good enough and automation good enough that most incidents were trivial to solve.  When we finally had to coordinate a full response to something that was entirely "weird" from a tech standpoint, the cracks in our armour were apparent.

  As mentioned in the common blockers section, we now practice weekly.  Every practice is followed with a post-mortem, and we iterate on both our engineering processes and incident response processes.  It has gone a long way to solve many of the other problem areas detailed in the original post (notably: Lack of knowledge, Organizational understanding, Badly defined roles).

  Practice really does drive improvement and I highly recommend it.

• 0
  
  Dan Goldberg March 23, 2020 23:07

  Good cautionary tale! I imagine others have had similar experiences.