Contents
|
The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.
Introduction
This article provides installation, configuration, and implementation details for the xMatters On-Demand for Splunk integration.
With the Splunk integration, you can:
- Get time critical information to the right person
- Get information to a user via push message, SMS, voice or email
- Pull event status, device delivery and response information into Splunk
Requirements
- Splunk 6.2
- xMatters On Demand
Integration Package
To begin, download the integration archive attached to this article and extract it to a location on your local machine. Some of the instructions in this article will reference specific folders within the extracted integration archive.
Configure xMatters
Before configuring anything in Splunk, you'll need to configure xMatters.
Create a REST API user
The first step in configuring xMatters is to create a REST API user to authenticate REST web service calls when injecting events.
This user needs to be able to work with events, but not update administrative settings. The best way to create a user for this integration is to have a dedicated "REST Web Service User" role that includes the permissions and capabilities. If this role does not exist in your deployment, you will need to create it, or ask Customer Support to create it for you. (For detailed procedures about creating the role, see Authentication and Permissions.) Alternatively, the "Full Access User" role can be used. This role has more permissions than needed, but will work until the "REST Web Service User" role is created.
In the following example, this role is named "REST Web Service User".
To create a REST API user:
- Log in to the target xMatters system.
- On the Users tab, click the Add New User icon.
- Enter the appropriate information for your new user.
- Assign the user the REST Web Service User role.
- Click Save.
- On the next page, set the web login ID and password.
Create users and groups that will receive notifications
To target users and groups, append "recipients: <targetName>" to the Alert Name. The python script will parse this out and add it to the payload to xMatters. <targetName> can be a user or a group and multiple entries can be added by separating them with commas (,).
For example:
My alert recipients: Database
My other alert recipients: bsmith,Database
- To create a new group, see Create a new group.
- To create a new user, see Add a new user.
Import the communication plan
The next step is to import the Splunk Communication Plan.
To import the Splunk communication plan:
- Download the attached .zip or tar.gz file to your system, and then extract the contents.
- In the target xMatters system, on the Developer tab, click Import Plan.
- Click Browse, and then locate the following file within the extracted integration archive:
- Click Import Plan.
- Once the communication plan has been imported, click Plan Disabled to enable the plan.
- In the Edit drop-down list, select Forms.
- For the Triggered Alert form, in the Not Deployed drop-down list, click Create Event Web Service.
- After you create the web service, the drop-down list label will change to Web Service Only.
- In the Web Service Only drop-down list, click Permissions.
- Enter the REST API user you created above, and then click Save Changes.
Accessing web service URLs
To get the web service URL for a form, in the Web Service Only drop-down list, click Access Web Service URL. Copy the highlighted URL at the top of the dialog box.
- Note: The Access Web Service URL option appears twice in the drop-down menu. Ensure that you click the option just below Create Event Web Service
Install and configure the integration
Now that you've configured xMatters, it's time to configure Splunk.
Install the package
The Splunk app is installed from the Splunkbase store.
To install the package:
Log in to your target Splunk web user interface and click the Find More Apps (the + button) from the Splunk home page.
After requesting a login to Splunkbase, the system will prompt for a restart.
Click Restart Now.
(If you choose to restart later, make sure you restart the Splunk server before using the xMatters Alerting app.)
Configure Splunk
To configure the Splunk integration:
After the restart, copy the $SPLUNK_HOME/etc/apps/xmatters_alert/bin/scripts/xMatters.py file to the $SPLUNK_HOME/bin/scripts directory.
Then log in to Splunk and navigate to the xMatters App configuration screen and enter the web service URL of your form and the other appropriate values from the xMatters configuration section, as explained in the table below.
Field Name | Field Value | Notes |
Username | splunk | The REST API user to authenticate to the Communication Plan form |
Password | <password> | The REST API user's password |
Web Service URL | https://<company>.<dc>.xmatters.com/reapi/2015-04-01/UUID-HERE/triggers | The Web Service URL copied from the Access Web Service URL button on the Triggered Alert form. |
Callbacks | status,deliveryStatus,response | If the Integration Agent is installed and configured, add the callbacks that should be sent to the IA when Splunk events are sent out. These should be one or all of status, deliveryStatus or response. Everything else will be ignored. |
Add an Alert:
To add an alert, make a search in Splunk and click Save As > Alert.
In the New Alert dialog, populate a name and make sure to append "recipients: <targetName>" to target the notification to a particular user or group.
Then, select Run a Script and populate the Filename field with "xMatters.py":
Install the integration agent (optional)
The integration agent can capture the event activity such as event status, device delivery and response information to a log on the Splunk server. This can then be pulled into Splunk for further processing. This is an optional step and is only needed to complete the two-way integration. If the agent is installed on a box remote from Splunk (such as when using Splunk in the cloud or architecture designs) be sure to install Splunks Universal Forwarder and have it index the splunk_callbacks.log file.
To install and configure the integration agent, see here, making sure to set the agent in "indirect" mode so that it will poll xMatters for information, rather than xMatters needing to push information to it. After the agent is successfully installed and connected to xMatters:
- Log in to xMatters and navigate to Developer > Event Domains.
- Click the applications link and scroll down to the Integration Services section.
- Click the Add New link.
- On the Integration Service Details page, in the Name field, type "splunk" and click Save.
Install the integration service
The last step is to install the integration service to the integration agent so that it is aware of the Splunk callbacks. To install this service:
- Copy the /components/integration-agent/splunk directory from the extracted integration archive file to the <IAHOME>/integrationservices/ directory.
- Open the <IAHOME>/conf/IAConfig.xml file and locate the <service-configs> node of the xml.
- Add the following line between the service-configs tags:
- Save the file and restart the integration agent.
- To verify it is configured correctly, log in to xMatters and navigate to the Developer tab > Event Domains > applications.
- Scroll to the Integration Services section.
- The "splunk" service should display as "Active":

- In Splunk, click Settings > Data Inputs.
- In the Files & Directories row, click Add New.
- Populate the /etc/integrationagent-5.1.5/log/splunk_callbacks.log in the File field and hit Next at the top of the screen.

- In the SourceType drop-down list on the left, select Structured > _json.
Test the integration
Test the integration to verify it works as expected.
To test the integration:
To test the integration, append the saved search string to an indexed log file. Using the Alert from above:
Which should fire an event into xMatters. Check the Reports tab in xMatters:
If the Integration Agent is installed and the Splunk integration service is ACTIVE, the $IA_HOME/log/splunk_callbacks.log file will be populated with the JSON callbacks and this can be searched in Splunk.
Troubleshooting
The $SPLUNK_HOME/var/log/splunk/xMatters.log file has detailed information about the $SPLUNK_HOME/bin/scripts/xMatters.py script. Any caught errors will be logged here.level: ERROR
level: DEBUG
2 Comments