Submit a Request

Change intelligence for digital services

Avatar
Karla Poplawski
Follow

Experts agree, change is the most common cause of incidents – the Google SRE Handbook says straight out: "Recent changes to a system can be a productive place to start identifying what's going wrong." Consider the following:

Those are big numbers! Clearly, visibility into recent changes is a key element of the service intelligence capabilities incident resolvers require to ensure your digital services continuously provide value to your customers.

Introducing change intelligence

Because digital services can experience thousands of changes per day, it’s critical to intelligently surface change information in a way that’s meaningful and actionable for resolvers.

With the latest enhancements to our digital operations platform, change information is embedded directly into the service intelligence capabilities of the xMatters incident management solution. By presenting relevant changes within the context of an incident, resolvers can identify recently changed services, gain greater insight into potential root causes, and immediately take action to mitigate and resolve the issue.

change-intelligence.png

Identify what's changed

During an incident, resolvers can pivot from the Incident Console to the service dependencies map to view the extent of the impact across their service landscape. We’ve enhanced this view with visibility into recent changes to services impacted by the incident, as well as the services they depend on.

Use the ‘Highlight Recent Changes’ check box and timeframe selector to set the change window. Any services with changes in that period will be highlighted with a blue badge on the map:

change-window.png

In the example above, although the Proxy service is identified as a potential root cause for the incident, this service depends on Customer Quota – which, although it doesn’t appear broken, has been recently changed.

Identify how it was changed

To see detailed change data for a service, click the options menu for a service and select ‘View Changes’:

view-changes-to-service.png

This opens a panel below the map with change records for the specific service, including:

  • The time when each change occurred, normalized to your time zone.
  • A summary of the change.
  • The type of change: Source Code, Deployment, Orchestration, Feature Toggle, Manual, or Other.
  • The source of the change.
  • An external change ID, if applicable.
  • Who or what made the change.

You can expand the panel to full screen, and click on any record in the list to view additional details that may have been provided about the change, in JSON, plain text, or another format:

change-details.png

Based on the change records in the example above, we’ve determined new code was deployed into production and there was a feature toggle change to Customer Quota. Even though the Proxy service is identified as a potential root cause of the incident, our insights into recent changes to our service landscape indicate that changes to Customer Quota may be the cause of the incident.

Take action

After you’ve identified the likely cause of the issue, you can take action directly from the service map by engaging subject matter experts as incident resolvers or by running automations that may be available for specific services.

In our example, we can click the Customer Quota service and select 'Notify to Engage' to pull in people from the team that owns that service as incident resolvers:

notify-to-engage.png

We can also select to run a rolling restart or regional failover from the Proxy service's list of available automations:

run-automation.png

How do I get change data into xMatters?

You’ll be able to feed change records into xMatters in several ways:

  • By adding the ‘Add Change Record’ step to your flows in Flow Designer.
  • Using one of our new low-code workflows or Flow Designer triggers designed to feed in changes from specific products, starting with GitLab, Azure Pipelines, BitBucket, GitHub, and LaunchDarkly.
  • Directly via the xMatters REST API.
  • Manually through the web user interface.

Availability

Change intelligence is included in our upcoming Pole Position Release:

  • Early Access Program: TBD
  • Non-production environments: Tuesday, June 21
  • Production environments: Tuesday, July 12

While change intelligence features in the Incident Console are available for Base and Advanced plans only, customers from all plan levels will have the ability to add change records to xMatters and browse service changes in the Service Catalog:

service-catalog-changes.png

 

Have more questions? Submit a request

3 Comments

Date Votes
  • 0
    Avatar
    Mike Lomax

    In my opinion, xMatters is not doing a very good job at attracting global customers when it comes to timestamps.  While some screens allow you to select the time zone the timestamps are listed in, others like the ones shown in this article as well as in the Workflow Activity screen do not.  This leaves the user guessing what time zone the timestamp relates to.  Is this the time sent in the trigger or the time zone based on my PC or what exactly?


    I love the ability to select what time zone you want to see timestamps listed in.  As an administrator who deals with systems located all around the world, this allows me to easily speak to on-site staff in times they are most familiar with.  At the very least, I wish that xMatters would at least include the time zone in all timestamps listed throughout the system.  Then us users would not be left guessing at what time the event happened and could speak more intelligently.


    BTW:  I have submitted multiple product enhancement requests around this topic and so far have not seen any changes to make this happen nor gotten any feedback as to why it won't happen.


    Please xMatters, do your global customers the favor of seriously considering these enhancements.  I know personally this is a roadblock to me recommending xMatters to any other global companies that are looking for a solution.


     


    Mike

  • 0
    Avatar
    Mike Lomax

    Doug,


    Yes...  That is what I am now and have been suggesting for years.  When the time zone is not included in the timestamps throughout, you are expecting the user to remember which time zone applies to each location where a timestamp appears.  Having either the "Display Time Zone" widget makes it clear that the time zone you are looking at is the one selected in that widget.


    I might add however, that on screens like when you drill into the "Recurrence" screen, you loose visibility of the "Display Time Zone" widget which requires you to remember what you had that widget set to.  I do find myself backing out to make sure I have the "Display Time Zone" widget set correctly for what I need in that case.  This usually happens when I have drilled into "Recurrence" without changing "Display Time Zone" widget from what I had it set to in a previous login session.


    The other struggle I have with time zones in xMatters is the fact that I live in AZ where we don't observe DST.  Most others in the company live where DST is observed.  If I set the times in xM using my personal account, even when I change "Display Time Zone" widget, and then a DST time change comes along, because my account does not observe DST, everyone that does has their schedules now off by 1 hour.  The only way I have found to deal with this is to consume another license to have a separate account that does observe DST so that I can set the times through that account.  xM really should fix this issue of not charge for the fact that I have to consume a license to work around the flaw.


    Don't get me wrong...  I love xMatters.  But these are issues that have always been a real pain and don't seem to be getting resolved.


    Mike

  • 0
    Avatar
    Doug Peete

    Hi Mike,


    Thank you for your feedback. The timezones on the change records are rendered in the timezone identified with your user profile. As an example, here is the screen in my default "US/Pacific" timezone:



    If I go into my profile and change my timezone to "Europe/London":



    The same view shows the timezones as:



    As you can see, we are implicitly speaking to the user in their timezone - so we are thinking about our global users, but we can always do better.


    The above behavior doesn't address the needs of users speaking to other people in terms of their timezone. It sounds like you've seen that we do that in the on-call scheduling where we implemented a "Display Time Zone" widget to quickly bounce between timezones without adjusting your profile. That is a use case that we heard from our customers. We can look to provide that sort of behavior in other places if that is what you are suggesting?


     

Please sign in to leave a comment.