Datadog Integration

Contents

Introduction

Prerequisites

Configure xMatters

Configure Datadog

Using the integration

Troubleshooting

Using basic authentication

Download resources

 The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.

Introduction

This article provides installation, configuration, and implementation details when integrating xMatters On-Demand with Datadog.

How it works

Datadog is a cloud monitoring solution that brings metrics from all of your apps, tools, and services into one place. By coupling Datadog with the power of xMatters, this integration:

  • Quickly identifies and notifies the on-call resource on a variety of devices
  • Allows for voice, SMS, and push messages to team members
  • Allows users to Mute and Claim alerts from Datadog Monitors directly from their device
  • Allows users to view xMatters shift data from within the Datadog interface

The integration uses custom webhooks in Datadog that can be @-mentioned in event or monitor alert messages, or even directly in a comment within the event stream. Each webhook sends the same payload to a different xMatters integration, which process the payload to create events in xMatters, or send comments back to the original Datadog event.

Features and updates 

This version of the xMatters On-Demand and Datadog integration uses the following custom webhooks created within Datadog.

Datadog Alerts

The Datadog Alerts feature creates a customized workflow within xMatters to handle Datadog monitored alert events. Whenever you @-mention the alerts webhook, xMatters will receive the payload and enrich the data via Datadog APIs.

Once the event is created within xMatters, status updates and delivery notifications can be sent back to the source event with Datadog via outbound integrations.

Engage with xMatters

The Engage with xMatters feature allows users to provide updates, request help, or start a conference bridge with colleagues from other teams to collaborate on events, event stream posts or snapshots within Datadog. 

Who's On Call

The Who's On Call feature displays current xMatters shift information within the Datadog event stream - no need to open a new browser window or login to xMatters.

For example, a DevOps team member wants to place a direct call to someone on the database team to confirm a solution. Instead of looking up the team in a company directory or calling a data center and hoping someone picks up, the DevOps person can use the Who's On Call webhook to see who is the right primary on-call person in the database group to contact. 

Check out our built-in integration

You can install a "built-in" version of this integration using the Integration Directory (Developer tab > Integrations). Built-in integrations are pre-configured for your xMatters: you don't need to download and import the communication plan, or follow the directions to configure xMatters as described below.

To configure this integration using the Integration Directory, see our online integration guide for Datadog. 

To continue setting up this packaged integration, use the following steps.

Prerequisites

There are a couple of tasks you can do ahead of time to make the configuration process a little easier.

Download the integration package

To begin, download the communication plan attached to this article; it contains everything needed for this integration. 

Note: Don't extract the contents of the .zip file; you'll import the file directly into xMatters.

Create API and application keys

For xMatters to authenticate with the Datadog API, you need to retrieve your API key, and create an application key for xMatters. You need these keys to set up communication between xMatters and Datadog.

To access the keys:

  1. Log into Datadog.
  2. Click Integrations > APIs.
    • You can find your API Key in the Key column of the API Keys list.
  3. To create a new Application Key for xMatters, type a name for the key (for example, "xMatters") in the New application key field, and click Create Application Key.
  4. You can now copy the application key from the Hash column in the Application Keys list.
(Datadog Reference)

Configure xMatters

To begin setting up your integration, configure the xMatters components.

Create an integration user

This integration requires a user who can authenticate REST web service calls when injecting events.

This user needs to be able to work with events, but does not need to update administrative settings. While you can use the default Company Supervisor role to authenticate REST calls, the best method is to create a user specifically for this integration with a dedicated role that includes the permissions and capabilities.

Note: If you're installing this integration into an xMatters trial instance, you don't need to create a new user. Instead, locate the "Integration User" sample user that was automatically configured with the REST Web Service User role when your instance was created and assign them a new password. You can then skip ahead to the next section.

To create an integration user:

  1. Log in to the target xMatters system.
  2. On the Users tab, click Add.
  3. Enter the appropriate information for your new user. Because this user affects how messages appear for recipients and how events are displayed in the reports and Communication Center, you may want to identify the user as specific to Datadog; for example:
    • First Name: Datadog
    • Last Name: Integration
    • User ID: datadog
  4. In the Roles area, add the REST Web Service User, Group Supervisor, People Supervisor and Support User roles.
  5. Make a note of the user credentials and details – you need them when configuring other parts of this integration.
  6. Click Add.

Create users and groups to receive notifications

The integration notifies groups and users when events are created in xMatters. 

To map xMatters users to Datadog, set the "Work Email" device for each user in xMattters to the same email as their handle within Datadog.  

Import the communication plan

The next step is to import the communication plan into xMatters.

To import the communication plan:

  1. In the target xMatters system, click Import Plan from the Developer tab.
  2. Click Choose File, and then locate the downloaded communication plan (.zip file).
  3. Click Import Plan.
    • Importing the plan automatically enables it, and enables its forms for web services.
  4. Click the Edit drop-down list for the plan, and select Access Permissions.
  5. Add the integration user you created above, and then click Save Changes.
  6. In the Edit drop-down list, select Forms.
  7. For the Conference Bridge form, click Web Service and select Sender Permissions.
  8. Enter the integration user you created above, and then click Save Changes.
  9. Repeat steps 6 to 8 for the remaining forms in the plan.

Set notification recipients

The Datadog Alert form uses the default recipients specified on the Layout tab to determine who to notify about an incoming alert.

  1. From the communication plan, click the Forms tab, and then click Edit > Layout beside the Datadog Alert form.
  2. In the Recipients section, add all of the groups and users you want to receive Datadog Alert notifications.
  3. Click Save Changes.

Configure endpoints

After you have imported the communication plan, you need to setup the Datadog endpoints.

  1. From the communication plan, click the Integration Builder tab, and then click Edit Endpoints.
  2. Select the xMatters endpoint, assign it to the integration user you created above (or, if you are using a trial instance, to "Integration User"), and then click Save Changes.
  3. Close the Endpoints dialog box.
  4. Click Edit Constants
  5. Select the Datadog API Key constant, paste your Datadog API key into the Value field, and then click Save Changes.
  6. Select the Datadog Application Key constant, paste your Datadog application key into the Value field, and then click Save Changes.
  7. Close the Constants dialog box.
Note: The current version of this integration does not support authentication on inbound integrations.

Configure inbound integrations

You need to the URL for each inbound integration (except the Create Datadog Event integration) to configure DataDog. 

To configure an inbound integration and retrieve its URL:

  1. Log out of xMatters, and then log back in as the integration user you configured above.
  2. Navigate to the Integration Builder tab for the communication plan, and expand the list of inbound integrations.
  3. Click the name of an integration.
  4. Scroll down to the bottom of the page, and click Copy URL beside the field:

CopyURL.jpg

Once you have retrieved the URLs, you can log the integration user out of xMatters. You can also set up Basic authentication using the integration user but there are a few more steps involved.

Configure Datadog

Now that you've configured xMatters to integrate with your system, it's time to configure Datadog to integrate with xMatters.

Note: The following sections require you to log into Datadog with an administrator account.

Create a webhook integration

  1. Log in to Datadog as an administrator, and then click Integrations > Integrations
  2. Locate and click the Webhooks badge.
  3. On the Configuration tab, click Add row link three times (you should have four empty entries).
  4. In the new rows, enter the following mappings into the Name and URL fields (see Configure inbound integrations for instructions on retrieving the URLs).
Name

URL (xMatters inbound integration) 

xmatters_alert Datadog Alert
xmatters_conference Conference Bridge
xmatters_engage Engage with xMatters
xmatters_on_call Who's on Call

Datadog_name-url-mappings.png
  1. Click Install Integration

Add an xMatters user

You can use an existing administrator account in Datadog, or create a specific user for xMatters to use when authenticating into Datadog to update tickets. 

To create a user in Datadog:

  1. In the bottom-left corner of the Datadog screen, click Team.
  2. In the Add teammates by email field, type a valid email address that is not currently in use in Datadog
  3. In the Invite As drop-down list, select Admin, and then click Invite.
    • Datadog sends an activation link to the specified address. 
  4. Open the email and click the link to activate the user.

When logged in as the user, you can enhance the integration by updating the user settings:

  1. In the bottom-left corner of the screen, click the user name.
  2. In the Name field, type a name to be displayed in the Events stream; e.g., "xMatters Integration".
  3. Click Save Profile.
  4. If the email address you used for this account will not be monitored, click the Preferences tab and disable all of the user's Email Subscriptions.

Using the integration

Datadog Alerts

You can test the Datadog Alerts feature by notifying the xmatters_alert custom webhook when one of your monitors moves into an Alert or Error state.

To send monitor alerts to xMatters:

  1. Click Monitors > Manage Monitors.
  2. Click the monitor with alerts you want to send to xMatters, and then click the Edit tab.
  3. In the Say what's happening section, type the message you want to send, and include the following to initiate the webhook:
@webhook-xmatters_alert
  1. Click Save.

datadog_xmatters-alert.png

Notes

  • Monitor bodies allow you to send different information depending on the transition of the monitor; for example, when transitioning to alert state or recovery state.
  • If you do not want to receive alerts when the monitor moves to a "No_Data" state, you may need to wrap the webhook @-mention in is_alert tags; i.e., {{#is_alert}} {{/is_alert}}
  • If you want the integration to automatically terminate xMatters events for recovered alerts, you will need to also wrap a webhook @-mention in {{#is_recovery}} {{/is_recovery}}
  • For more details on message template variables, consult the Datadog documentation.  

Engage with xMatters

You can test the Engage with xMatters feature using the Event Stream or a Snapshot Comment.

To engage with xMatters:

  1. Click Events.
  2. In the Leave a status update field for an event, use the following format to initiate the webhook (see the Notes below for more syntax information):
<message_text> @webhook-xmatters_engage <recipient>
  1. Click Post to create the post in Datadog and trigger the event in xMatters.
    • If enabled, outbound integrations for status updates, delivery notifications and response updates create comments and other updates on the Datadog post.
datadog_engage_post.png

Notes

Engage with xMatters posts have some important required formatting:

  • All text preceding the @-mention is used as the message when sending notifications to the recipients.
  • All text after the @-mention must identify the recipients. For example, in the screenshot above, the "Database" and "Hardware" xMatters groups are targeted for notification.
  • You can use the same format to initiate the xmatters_conference webhook. 
  • If you want to specify an external conference bridge for an xmatters_conference webhook, add -bridge <bridge_name>, where <bridge_name> is a valid conference bridge already configured in xMatters. For example:
Please join the conference @webhook-xmatters_conference Database Hardware -bridge "My External Conference"

Who's On Call

When using the Who's On Call feature, you can use two different post types.

To test the Who's On Call feature:

  1. Click Events.
  2. In the Leave a status update field for an event, use the following format to initiate the webhook (see the Notes below for more syntax information):
@webhook-xmatters_on_call <group>
  1. Click Post.
    • xMatters displays the Who's On Call results.

Notes

  • Any text after the @-mention determines the type of Who's On Call you want to make.
  • To get a group listing, leave the text after the @-mention empty or add -page x where x is a positive integer. The xMatters integration replies with the page of groups for the value of x or, if empty, the first page of groups:

datadog_on-call_groups.png

  • To get on call shift details for a specific group, enter the name of a valid xMatters group after the @-mention. The xMatters integration replies with all active shifts for that group and the team members who are on call:

datadog_on-call.png

Troubleshooting

There are several places you can inspect when troubleshooting why an event is not sent to xMatters. 

Inbound to xMatters

First, in Datadog, any errors in connecting to the URL encountered by the webhook will create new events in the Datadog Event Stream. For example:

datadog_unauthorized-message.png

The failing webhook is highlighted in the payload, indicating that you are not authorized to access the @webhook-xmatters_engage feature.

The next place to look is the Integration Builder Activity Stream. While on the Integration Builder tab, expand the Inbound Integrations section, click the gear icon beside the intended integration service, and then click Activity Stream.

The Activity Stream contains the incoming (and for outbound integrations, the outgoing) request, any logging statements as well as the final event creation messages.   

Outbound from xMatters

For the two-way integration from xMatters to Datadog, the primary source of logging is the Integration Builder Activity Stream for that particular integration.

For example, if you are troubleshooting why a ticket was not assigned to a user after they replied with "Assign to me", check the Activity Steam for the Datadog Alert Outbound Response outbound integration:

datadog_activity_stream.png

Extend your integration: using basic authentication

When the communication plan is imported into xMatters, the Integration Builder services are set to use URL authentication. You can choose to use basic authentication instead so the endpoints cannot be triggered unless the calling system authenticates with a user name and password – the integration user, in this case. (As an added bonus, using basic authentication means you don't have to log out and log back in to configure the inbound integrations!)

Enable basic authentication

First you need to enable basic authentication for each inbound integration (except the Create Datadog Event integration).

To set up your inbound integrations for basic authentication:

  1. In the Integration Builder, expand the list of inbound integrations.
  2. Click the name of an integration to view its details.
  3. Under the Select authentication method step, select Basic from the drop-down list.
  4. Click Update Integration.
  5. Scroll down to the bottom of the page, and click Copy URL beside the field (you'll need this later to configure DataDog):

CopyURL.jpg

Encode the authentication header value

All webhooks within Datadog need a custom authentication header that identifies the generated integration credentials to access the xMatters inbound integration.

To create this value:

  1. Go to https://www.base64encode.org/.
  2. In the “Encode to Base64 format” area, type the username and password of the integration user in the following format: username:password
    • For example: datadog:5x>W!AkdfaHJd6092D
  3. Click Encode, and copy the result.

Add the custom headers to the Datadog webhooks

  1. Log in to Datadog as an administrator, and then click Integrations > Integrations
  2. Locate and click the Webhooks badge.
  3. For each xMatters webhook, add the following text to the Headers field, replacing <authentication_header_value> with the encoded result from the previous step.
{
"Authorization": "Basic <authentication_header_value>"
}

Example:

datadog_auth_headers.png

Download resources

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk