Contents Introduction Configure xMatters Create a REST API user Create users and groups that will receive notifications Import the Splunk communication plan Install and configure the integration Install the package Configure Splunk Field descriptions Install the Integration Agent Install the integration service Add the callback log data source Test the integration Troubleshooting Download resources

The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.

Introduction

This article provides installation, configuration, and implementation details for the xMatters On-Demand for Splunk integration.

With the Splunk integration, you can:

• Get time critical information to the right person
• Get information to a user via push message, SMS, voice or email
• Pull event status, device delivery and response information into Splunk

Requirements

• Splunk 6.2
• xMatters On Demand

Integration Package

To begin, download the integration archive attached to this article and extract it to a location on your local machine. Some of the instructions in this article will reference specific folders within the extracted integration archive.

Configure xMatters

Before configuring anything in Splunk, you'll need to configure xMatters. 

Create a REST API user

The first step in configuring xMatters is to create a REST API user to authenticate REST web service calls when injecting events.

This user needs to be able to work with events, but not update administrative settings. The best way to create a user for this integration is to have a dedicated "REST Web Service User" role that includes the permissions and capabilities. If this role does not exist in your deployment, you will need to create it, or ask Client Assistance to create it for you. (For detailed procedures about creating the role, see Authentication and Permissions.) Alternatively, the "Full Access User" role can be used. This role has more permissions than needed, but will work until the "REST Web Service User" role is created.

In the following example, this role is named "REST Web Service User".

To create a REST API user:

1. Log in to the target xMatters system.
2. On the Users tab, click the Add New User icon.
3. Enter the appropriate information for your new user.
4. Assign the user the REST Web Service User role.
5. Click Save.
6. On the next page, set the web login ID and password. 

Make a note of these details; you will need them when configuring the Splunk side of the integration.

Create users and groups that will receive notifications

To target users and groups, append "recipients: <targetName>" to the Alert Name. The python script will parse this out and add it to the payload to xMatters. <targetName> can be a user or a group and multiple entries can be added by separating them with commas (,). 

For example:

My alert recipients: Database
My other alert recipients: bsmith,Database

• To create a new group, see Create a new group.
• To create a new user, see Add a new user.

You can also create multiple groups and users at once using the EPIC feature. 

Import the communication plan

The next step is to import the Splunk Communication Plan.

To import the Splunk communication plan:

1. Download the attached .zip or tar.gz file to your system, and then extract the contents.
2. In the target xMatters system, on the Developer tab, click Import Plan.
3. Click Browse, and then locate the following file within the extracted integration archive:

/components/xmatters/Splunk.zip

4. Click Import Plan.
5. Once the communication plan has been imported, click Plan Disabled to enable the plan.
6. In the Edit drop-down list, select Forms.
7. For the Triggered Alert form, in the Not Deployed drop-down list, click Create Event Web Service.
   • After you create the web service, the drop-down list label will change to Web Service Only.
8. In the Web Service Only drop-down list, click Permissions.
9. Enter the REST API user you created above, and then click Save Changes.

Accessing web service URLs

To get the web service URL for a form, in the Web Service Only drop-down list, click Access Web Service URL. Copy the highlighted URL at the top of the dialog box.

• Note: The Access Web Service URL option appears twice in the drop-down menu. Ensure that you click the option just below Create Event Web Service

You'll need these URLs when you configure the Splunk side of the integration.

Install and configure the integration

Now that you've configured xMatters, it's time to configure Splunk. 

Install the package

The Splunk app is installed from the Splunkbase store. 

To install the package:

Log in to your target Splunk web user interface and click the Find More Apps (the + button) from the Splunk home page. 

 Search for "xMatters", and then click the Install free button.

 

After requesting a login to Splunkbase, the system will prompt for a restart.

Click Restart Now.

(If you choose to restart later, make sure you restart the Splunk server before using the xMatters Alerting app.)

Configure Splunk

To configure the Splunk integration:

After the restart, copy the $SPLUNK_HOME/etc/apps/xmatters_alert/bin/scripts/xMatters.py file to the $SPLUNK_HOME/bin/scripts directory. 

Then log in to Splunk and navigate to the xMatters App configuration screen and enter the web service URL of your form and the other appropriate values from the xMatters configuration section, as explained in the table below. 

Field Name	Field Value	Notes
Username	splunk	The REST API user to authenticate to the Communication Plan form
Password	<password>	The REST API user's password
Web Service URL	https://<company>.<dc>.xmatters.com/reapi/2015-04-01/UUID-HERE/triggers	The Web Service URL copied from the Access Web Service URL button on the Triggered Alert form.
Callbacks	status,deliveryStatus,response	If the Integration Agent is installed and configured, add the callbacks that should be sent to the IA when Splunk events are sent out. These should be one or all of status, deliveryStatus or response. Everything else will be ignored.

Add an Alert:

To add an alert, make a search in Splunk and click Save As > Alert.  

 

In the New Alert dialog, populate a name and make sure to append "recipients: <targetName>" to target the notification to a particular user or group. 

 

Then, select Run a Script and populate the Filename field with "xMatters.py":

 

Install the integration agent (optional)

The integration agent can capture the event activity such as event status, device delivery and response information to a log on the Splunk server. This can then be pulled into Splunk for further processing. This is an optional step and is only needed to complete the two-way integration. If the agent is installed on a box remote from Splunk (such as when using Splunk in the cloud or architecture designs) be sure to install Splunks Universal Forwarder and have it index the splunk_callbacks.log file. 

To install and configure the integration agent, see here, making sure to set the agent in "indirect" mode so that it will poll xMatters for information, rather than xMatters needing to push information to it. After the agent is successfully installed and connected to xMatters:

1. Log in to xMatters and navigate to Developer > Event Domains.
2. Click the applications link and scroll down to the Integration Services section.
3. Click the Add New link.
4. On the Integration Service Details page, in the Name field, type "splunk" and click Save. 

 

Install the integration service

The last step is to install the integration service to the integration agent so that it is aware of the Splunk callbacks. To install this service:

1. Copy the /components/integration-agent/splunk directory from the extracted integration archive file to the <IAHOME>/integrationservices/ directory. 
2. Open the <IAHOME>/conf/IAConfig.xml file and locate the <service-configs> node of the xml.
3. Add the following line between the service-configs tags:

<path>splunk/splunk.xml</path>

4. Save the file and restart the integration agent. 
5. To verify it is configured correctly, log in to xMatters and navigate to the Developer tab > Event Domains > applications.
6. Scroll to the Integration Services section.
• The "splunk" service should display as "Active":

 

Add the callback log data source

Note, if using a box remote from Splunk be sure to install and configure the Universal Forwarder. 

 

Now add the callbacks to the Splunk indexes:

1. In Splunk, click Settings > Data Inputs.
2. In the Files & Directories row, click Add New.
3. Populate the /etc/integrationagent-5.1.5/log/splunk_callbacks.log in the File field and hit Next at the top of the screen. 

4. In the SourceType drop-down list on the left, select Structured > _json.
   
   

Test the integration

Test the integration to verify it works as expected. 

To test the integration:

To test the integration, append the saved search string to an indexed log file. Using the Alert from above:

Which should fire an event into xMatters. Check the Reports tab in xMatters:

 

If the Integration Agent is installed and the Splunk integration service is ACTIVE, the $IA_HOME/log/splunk_callbacks.log file will be populated with the JSON callbacks and this can be searched in Splunk. 

Troubleshooting

The $SPLUNK_HOME/var/log/splunk/xMatters.log file has detailed information about the $SPLUNK_HOME/bin/scripts/xMatters.py script. Any caught errors will be logged here. 

 

Additionally, the $SPLUNK_HOME/etc/apps/xmatters_alert/bin/scripts/xMConfig.cfg file contains a logging level that can be set to DEBUG for more detailed information. 

To enable debugging, open the $SPLUNK_HOME/etc/apps/xmatters_alert/bin/scripts/xMConfig.cfg file in a text editor:

 

[Logging]
level: ERROR

 

Change this to:

 

[Logging]
level: DEBUG

 

And then reproduce the issue and check the $SPLUNK_HOME/var/log/splunk/xMatters.log file for details. 

 

 

Download resources

Splunk integration package - zip	Download
Splunk integration package - tar.gz	Download