Splunk Integration v1.0

Introduction

 

Splunk Enterprise is the industry-leading platform for operational intelligence. Collect and index any machine data from virtually any source in real time. Search, monitor, analyze and visualize your data to gain new insights and intelligence.

 

Combining Splunk with the power of xMatters On-Demand enables customers to receive immediate notifications to the appropriate on-call response teams to reduce the time to respond to critical incidents.

Integration Overview

 

The xMatters On-Demand & Splunk integration leverages the Alert Script functionality provided by Splunk. This allows for a python script to be called when an Alert is triggered. The script references the xMConfig.cfg file to retrieve the username, password and endpoint and builds a payload to send to xMatters On-Demand

 

When the Alert is triggered, Splunk calls the script specified in the Alert and passes the following items to xMatters:

  • SPLUNK_ARG_0 Script name
  • SPLUNK_ARG_1 Number of events returned
  • SPLUNK_ARG_2 Search terms
  • SPLUNK_ARG_3 Fully qualified query string
  • SPLUNK_ARG_4 Name of report
  • SPLUNK_ARG_5 Trigger reason (for example, "The number of events was greater than 1")
  • SPLUNK_ARG_6 Browser URL to view the report
  • SPLUNK_ARG_7 Not used for historical reasons
  • SPLUNK_ARG_8 File in which the results for this search are stored (contains raw results)

How to configure xMatters On-Demand

 Create the Splunk user

 

You will need to create a new user in xMatters On-Demand which will be used to send the Splunk notifications. Navigate to the Users tab and click the Add User button in the upper-right corner: 

 

 

Enter the appropriate details; make sure you assign the "Standard User" role to your new user:

 

  

Click Save, and then create a web login ID and password:

 

 

Create an on-call group to receive Splunk alerts

 

The integration will target an on-call group designated for responding to Splunk alerts. To create this group, navigate to the Groups tab, and then click the Add Group in the upper-right corner:

 

 

In the Add Group dialog, enter a name, description and some members. This will be a 24x7 group; other details, such as coverages and rotations, can be added later.

 

Click Add Group to finish creating the group:

 

  

Import the Splunk Communication Plan

 

First, download and extract the integration package appropriate for your operating system; you can find links to the integration packages at the bottom of this page.

 

In xMatters, navigate to the Developer tab, and then click Import Plan.

 

 

Click Browse, select the xM-Splunk_1_0/components/xmatters/Splunk.zip file from the extracted integration package, and then click Import Plan. 

 

 

 

Enable the plan and form, and create the web service when you are ready to start sending alerts:

 

 

Add the Splunk user you created to the list of permitted form initiators.

  

 

 

Next, configure the default recipients in the form layout:

 

In the recipients section, add the group that you want to receive alerts from Splunk:

 

 

Finally, copy the web service URL (endpoint) for use in the Splunk configuration: click the Web Service Only drop down, and then select Access Web Service URL:

 

 

 

How to configure Splunk

Configure the xMatters alert script

First, extract the alert script files from the zip file from xM-Splunk_1_0/components/splunk and copy them all to SPLUNK_HOME/bin/scripts.

 

There are 3 files:

  • xMatters.py: The script that makes the REST call to xMatters
  • xMConfig.cfg: The config file containing the username, password, REST endpoint and logging details
  • sf-class2-root.crt: The Starfield root certificate.

 

Once the files are copied, open the xMConfig.cfg file in your favorite text editor and update the username and password with the Splunk user information set above. Then paste in the url copied and save the file. 

 

 

Create Triggered Alert

Login to the Splunk UI and enter a search term. In this case we are searching for entries of xMatters:

 

 

After clicking the search button, the results are displayed. Click Save As > Alert to save this search as an alert:

 

 

This will display the Save As Alert dialog. Enter a name (the name of this alert will be sent to recipients via email, so it should be something descriptive) and set any other relevant details:

 

 

Click the Next button to display the actions dialog. Select the Run a Script check box and enter the name of the xMatters script (xMatters.py) that was copied in the steps above. Click Save. 

 

 

Troubleshooting 

Test the Integration

 

Testing the integration will depend on the search terms of the alert. In our lab environment, the /var/log directory is indexed by Splunk, so simply adding the Search Term to a new log file will trigger the alert:

 

 

Debugging

 

The SPLUNK_HOME/bin/scripts/xMatters.py python script reads the SPLUNK_HOME/bin/scripts/xMConfig.cfg file for logging details. The logging library is similar to log4j and allows for ERROR, INFO, and DEBUG levels. The name of the log file is set in this config file and can be found in the SPLUNK_HOME/var/log directory. 

 

For troubleshooting just the script, call the following command from the SPLUNK_HOME directory to simulate an Alert from Splunk:

 

/opt/splunk/bin/splunk cmd python '/opt/splunk/bin/scripts/xMatters.py' '1' 'xMatters' 'xMatters' 'xMatters Alert' 'Saved Search [xMatters Alert] always(1)' 'http://splunk:8000/app/search/search?q=%7Cloadjob%20rt_scheduler__admin__search__RMD5dfb8f0a8de8a530d_at_1413836990_0.7%20%7C%20head%201%20%7C%20tail%201&earliest=0&latest=now' '' '/opt/splunk/var/run/splunk/dispatch/rt_scheduler__admin__search__RMD5dfb8f0a8de8a530d_at_1413836990_0.7/per_result_alert/tmp_0.csv.gz'

Alternatively, the SPLUNK_HOME/var/log/splunkd.log and SPLUNK_HOME/var/log/python.log contain the python command. However, I had issues running just /opt/splunk/bin/python <scriptname> .... so running the /bin/splunk cmd python <scriptname> is recommended.

 

Downloads

 

Splunk Integration Package - zip Download (5.8 KB)
Splunk Integration Package - tar.gz Download (5.8 KB)
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk