Datadog Integration

Contents

Introduction

Prerequisites

Configure xMatters

Configure Datadog

Using the integration

Troubleshooting

Appendix A - Authenticating via the Integration Builder

Download resources

 The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.

Introduction

This article provides installation, configuration, and implementation details when integrating xMatters On-Demand with Datadog.

How it works

Datadog is a cloud monitoring solution that brings metrics from all of your apps, tools, and services into one place. By coupling Datadog with the power of xMatters, this integration:

  • Quickly identifies and notifies the on-call resource on a variety of devices
  • Allows for voice, SMS, and push messages to team members
  • Allows users to Mute and Claim alerts from Datadog Monitors directly from their device
  • Allows users to view xMatters shift data from within the Datadog interface

The integration uses custom webhooks in Datadog that can be @-mentioned in event or monitor alert messages, or even directly in a comment within the event stream. Each webhook sends the same payload to a different xMatters integration, which process the payload to create events in xMatters, or send comments back to the original Datadog event.

Features and updates 

This version of the xMatters On-Demand and Datadog integration uses the following custom webhooks created within Datadog.

Datadog Alerts

The Datadog Alerts feature creates a customized workflow within xMatters to handle Datadog monitored alert events. Whenever you @-mention the alerts webhook, xMatters will receive the payload and enrich the data via Datadog APIs.

Once the event is created within xMatters, status updates and delivery notifications can be sent back to the source event with Datadog via outbound integrations.

Engage with xMatters

The Engage with xMatters feature allows users to provide updates, request help, or start a conference bridge with colleagues from other teams to collaborate on events, event stream posts or snapshots within Datadog. 

Who's On Call

The Who's On Call feature displays current xMatters shift information within the Datadog event stream - no need to open a new browser window or login to xMatters.

For example, a DevOps team member wants to place a direct call to someone on the database team to confirm a solution. Instead of looking up the team in a company directory or calling a data center and hoping someone picks up, the DevOps person can use the Who's On Call webhook to see who is the right primary on-call person in the database group to contact. 

A note to xMatters trial users

If you are using the trial version of xMatters, you can install a "built-in" version of this integration using the Integration Directory (Developer tab > Integrations). Built-in integrations are pre-configured for your xMatters: you don't need to download and import the communication plan, or follow the directions to configure xMatters as described below.
 
Note: You will need the API and application keys from Datadog, as described in the following Prerequisites section, but you can create these keys at any time.

To continue configuring this integration for your trial version of xMatters, skip ahead to Configure Datadog.

Prerequisites

There are a couple of tasks you can do ahead of time to make the configuration process a little easier.

Download the integration package

To begin, download the communication plan attached to this article; it contains everything needed for this integration. 

Note: You do not need to extract the contents of the .zip file; you will be importing the file directly into xMatters.

Create API and application keys

For xMatters to authenticate with the Datadog API, you need to retrieve your API key, and create an application key for xMatters. You will need these keys to set up communication between xMatters and Datadog.

To access the keys:

  1. Log into Datadog.
  2. Click Integrations > APIs.
    • You can find your API Key in the Key column of the API Keys list.
  3. To create a new Application Key for xMatters, type a name for the key (for example, "xMatters") in the New application key field, and click Create Application Key.
  4. You can now copy the application key from the Hash column in the Application Keys list.
(Datadog Reference)

Configure xMatters

To begin setting up your integration, configure the xMatters components.

Create an integration user

This integration requires an xMatters user with the correct permissions to authenticate REST web service calls when injecting events.

This user needs to be able to work with events, but does not need to update administrative settings. While you can use the default Company Supervisor role to authenticate REST calls, the best method is to create a user specifically for this integration with a dedicated role that includes the permissions and capabilities.

NOTE: To successfully execute some of the tasks for this integration, the integration user requires some additional permissions in xMatters, as indicated in the following steps.

To create a REST API user:

  1. Log in to the target xMatters system.
  2. On the Users tab, click Add.
  3. Enter the appropriate information for your new user. Because this user will affect how messages appear for recipients and how events will be displayed in the reports and Communication Center, you may want to identify the user as specific to Datadog; for example:
    • First Name: Datadog
    • Last Name: Integration
    • User ID: datadog
  4. In the Roles area, add the REST Web Service User, Group Supervisor, People Supervisor and Support User roles.
    • If the REST Web Service User role does not exist in your deployment already, you may need to ask your xMatters Client Success Manager to create it for you. 
  5. Click Add.
Make a note of the user credentials and details; you will need them when configuring other parts of this integration.

Create users and groups that will receive notifications

The integration will notify groups and users when events are created in xMatters.

To map xMatters users to Datadog, set the "Work Email" device for each user in xMattters to the same email as their handle within Datadog.  

Import the communication plan

The next step is to import the communication plan into xMatters.

To import the communication plan:

  1. In the target xMatters system, click Import Plan from the Developer tab.
  2. Click Browse, and then locate the downloaded communication plan (.zip file).
  3. Click Import Plan.
    • Importing the plan will automatically enable it, and enable its forms for web services.
  4. Once the communication plan has been imported, click Edit > Forms.
  5. For the Conference Bridge form, click Web Service Only > Sender Permissions.
  6. Enter the REST API user you created above, and then click Save Changes.
  7. Repeat steps 5 and 6 for the remaining forms in the plan.

Set notification recipients

The Datadog Alert form uses the default recipients specified on the Layout tab to determine who to notify about an incoming alert.

  1. From the communication plan, click the Forms tab, and then click Edit > Layout in the Datadog Alert form row.
  2. In the Recipients section, add all of the groups and users you want to receive Datadog Alert notifications.
  3. Click Save Changes.

Configure integration settings

After you have imported the communication plan, you need to setup the xMatters and Datadog endpoints.

  1. Return to the Datadog communication plan (if you're on the Layout tab, click the Datadog 1.0 link at the top of the tab), click the Integration Builder tab, and then click Edit Endpoints.
  2. Select the xMatters endpoint, assign it to the Datadog REST API user you created above, and then click Save Changes.
  3. Close the Endpoints dialog box.
  4. Click Edit Constants
  5. Select the Datadog API Key constant, paste your Datadog API key into the Value field, and then click Save Changes.
  6. Select the Datadog Application Key constant, paste your Datadog application key into the Value field, and then click Save Changes.
  7. Close the Constants dialog box.
Note: The current version of this integration does not support authentication on inbound integrations.

Accessing URLs

Each integration service has its own URL that you can use to target it from Datadog.

To get the URL for an integration service:

  1. On the Integration Builder tab, expand the list of inbound integrations.
  2. Click the gear icon beside the integration service you want to target, and then select Integration URL.

  1. Copy the URL displayed in the dialog box:

 

With the exception of the Create Datadog Event integration, you will need the URL for each inbound integration service when you configure Datadog.

Configure Datadog

Now that you've configured xMatters to integrate with your system, it's time to configure Datadog to integrate with xMatters.

Note: The following sections require you to log into Datadog with an administrator account.

Create a webhook integration

  1. Log in to Datadog as an administrator, and then click Integrations > Integrations
  2. Locate and click the Webhooks badge.
  3. On the Configuration tab, click Add row link three times (you should have four empty entries).
  4. In the new rows, enter the following mappings into the Name and URL fields (for instructions on retrieving the Integration URLs see the Accessing URLs section, above).
    • For trial users, these URLs are available on the Datadog configuration screen, after you save your initial configuration.
Name

URL (xMatters inbound integration) 

xmatters_alert Datadog Alert
xmatters_conference Conference Bridge
xmatters_engage Engage with xMatters
xmatters_on_call Who's on Call


  1. Click Install Integration

Add an xMatters user

You can use an existing administrator account in Datadog, or create a specific user for xMatters to use when authenticating into Datadog to update tickets. 

To create a user in Datadog:

  1. In the bottom-left corner of the Datadog screen, click Team.
  2. In the Add teammates by email field, type a valid email address that is not currently in use in Datadog
  3. In the Invite As drop-down list, select Admin, and then click Invite.
    • Datadog will send an activation link to the specified address. 
  4. Open the email and click the link to activate the user.

When logged in as the user, you can enhance the integration by updating the user settings:

  1. In the bottom-left corner of the screen, click the user name.
  2. In the Name field, type a name to be displayed in the Events stream; e.g., "xMatters Integration".
  3. Click Save Profile.
  4. If the email address you used for this account will not be monitored, click the Preferences tab and disable all of the user's Email Subscriptions.

Using the integration

Datadog Alerts

You can test the Datadog Alerts feature by notifying the xmatters_alert custom webhook when one of your monitors moves into an Alert or Error state.

To send monitor alerts to xMatters:

  1. Click Monitors > Manage Monitors.
  2. Click the monitor with alerts you want to send to xMatters, and then click the Edit tab.
  3. In the Say what's happening section, type the message you want to send, and include the following to initiate the webhook:
@webhook-xmatters_alert
  1. Click Save.

Notes

  • Monitor bodies allow you to send different information depending on the transition of the monitor; for example, when transitioning to alert state or recovery state.
  • If you do not want to receive alerts when the monitor moves to a "No_Data" state, you may need to wrap the webhook @-mention in is_alert tags; i.e., {{#is_alert}} {{/is_alert}}
  • If you want the integration to automatically terminate xMatters events for recovered alerts, you will need to also wrap a webhook @-mention in {{#is_recovery}} {{/is_recovery}}
  • For more details on message template variables, consult the Datadog documentation.  

Engage with xMatters

You can test the Engage with xMatters feature using the Event Stream or a Snapshot Comment.

To engage with xMatters:

  1. Click Events.
  2. In the Leave a status update field for an event, use the following format to initiate the webhook (see the Notes below for more syntax information):
<message_text> @webhook-xmatters_engage <recipient>
  1. Click Post to create the post in Datadog and trigger the event in xMatters.
    •  If enabled, outbound integrations for status updates, delivery notifications and response updates will create comments and other updates on the Datadog post.

Notes

Engage with xMatters posts have some important required formatting:

  • All text preceding the @-mention is used as the message when sending notifications to the recipients.
  • All text after the @-mention must identify the recipients. For example, in the screenshot above, the "Database" and "Hardware" xMatters groups are targeted for notification.
  • You can use the same format to initiate the xmatters_conference webhook. 
  • If you want to specify an external conference bridge for an xmatters_conference webhook, add -bridge <bridge_name>, where <bridge_name> is a valid conference bridge already configured in xMatters. For example:
Please join the conference @webhook-xmatters_conference Database Hardware -bridge "My External Conference"

Who's On Call

When using the Who's On Call feature, you can use two different post types.

To test the Who's On Call feature:

  1. Click Events.
  2. In the Leave a status update field for an event, use the following format to initiate the webhook (see the Notes below for more syntax information):
@webhook-xmatters_on_call <group>
  1. Click Post.
    • xMatters will display the Who's On Call results.

Notes

  • Any text after the @-mention determines the type of Who's On Call you want to make.
  • To get a group listing, leave the text after the @-mention empty or add -page x where x is a positive integer. The xMatters integration will reply with the page of groups for the value of x or, if empty, the first page of groups:

 

  • To get on call shift details for a specific group, enter the name of a valid xMatters group after the @-mention. The xMatters integration will reply with all active shifts for that group and the team members who are on call:

Troubleshooting

There are several places you can inspect when troubleshooting why an event is not sent to xMatters. 

Inbound to xMatters

First, in Datadog, any errors in connecting to the URL encountered by the webhook will create new events in the Datadog Event Stream. For example:

The failing webhook is highlighted in the payload, indicating that you are not authorized to access the @webhook-xmatters_engage feature.

The next place to look is the Integration Builder Activity Stream. While on the Integration Builder tab, expand the Inbound Integrations section, click the gear icon beside the intended integration service, and then click Activity Stream.

The Activity Stream contains the incoming (and for outbound integrations, the outgoing) request, any logging statements as well as the final event creation messages.   

Outbound from xMatters

For the two-way integration from xMatters to Datadog, the primary source of logging is the Integration Builder Activity Stream for that particular integration.

For example, if you are troubleshooting why a ticket was not assigned to a user after they replied with "Assign to me", check the Activity Steam for the Response Callback outbound integration:

Appendix A - Authenticating via the Integration Builder

Enable authentication

When the communication plan is imported into xMatters, the Integration Builder services do not have authentication configured. By enabling authentication for the integrations, the endpoints cannot be triggered unless the caller authenticates with a user name and password.

To enable authentication, expand the list of inbound integrations, and then click Authentication Off. The Authentication dialog will display a randomly generated username and password. Make a note of these values (or copy them to a temporary text file). They will not be displayed again, but you will need them when configuring Datadog, as described below.

Encode the authentication header value

All webhooks within Datadog need a custom authentication header that identifies the generated integration credentials to access the xMatters inbound integration.

To create this value:

  1. Go to https://www.base64encode.org/.
  2. In the “Encode to Base64 format” area, type the username and password randomly generated in the previous step in the following format: username:password
    • For example: 09803425lknhalkzl37120:5x>W!AkdfaHJd6092D
  3. Click Encode, and copy the result.

Add the custom headers to the Datadog webhooks

  1. Log in to Datadog as an administrator, and then click Integrations > Integrations
  2. Locate and click the Webhooks badge.
  3. For each xMatters webhook, add the following text to the Headers field, replacing <authentication_header_value> with the encoded result from the previous step.
{
"Authorization": "Basic <authentication_header_value>"
}

Example:

 

Download resources

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk