Splunk Alert Actions

The information in this article is the intellectual property of xMatters and is intended only for use with xMatters products by xMatters customers and their employees. Further, this intellectual property is proprietary and must not be reused or resold.

Introduction

By coupling the operational intelligence power of Splunk with the alert intelligence of xMatters, you can:

  • Get time critical information to the right person
  • Get information to a user via push message, SMS, voice or email

This article details the steps to install the xMatters Alert Action for Splunk.

Requirements

  • Splunk 6.3
    • On premise or Splunk Cloud*
  • xMatters On Demand 5.5.86

Note: Contact Splunk support to install the integration in Splunk Cloud. 

Configure xMatters

Create a REST API user

This integration requires a REST API user to authenticate REST web service calls when injecting events.

This user needs to be able to work with events, but not update administrative settings. The best way to create a user for this integration is to have a dedicated “REST Web Service User” role that includes the permissions and capabilities. If this role does not exist in your deployment, you will need to create it, or ask your xMatters Client Success Manager to create it for you. (For detailed procedures about creating the role, see Authentication and Permissions.)

In the following example, this role is named “REST Web Service User”.

To create a REST API user:

  1. Log in to the target xMatters system.
  2. On the Users tab, click the Add New User icon.
  3. Enter the appropriate information for your new user.
  4. Assign the user to the REST Web Service User role.
  5. Click Save.
  6. On the next page, set the web login ID and password.
Make a note of these details; you will need them when configuring other parts of this integration.

Create users and groups to target

To create a new group, see Create a new group.
To create a new user, see Add a new user.
You can create multiple groups and users at once using the EPIC feature.

Import the communication plan

Download the communication plan (.zip file) attached to this article, but do not extract the contents; you can import the archive directly into xMatters.

To import the communication plan:

  1. In the target xMatters system, on the Developer tab, click Import Plan.
  2. Click Browse, and then locate the downloaded .zip file.
  3. Click Import Plan.
  4. Once the communication plan has been imported, click Plan Disabled to enable the plan.
  5. In the Edit drop-down list, select Forms.
  6. For the Splunk Alert Actions form, in the Not Deployed drop-down list, click Create Event Web Service.
    • After you create the web service, the drop-down list label will change to Web Service Only.
  7. In the Web Service Only drop-down list, click Permissions.
  8. Enter the REST API user you created above, and then click Save Changes.

Accessing web service URLs

To get the web service URL for a form, in the Web Service Only drop-down list, click Access Web Service URL. Copy the highlighted URL at the top of the dialog box.

    Note: The Access Web Service URL option may appear twice in the drop-down menu. Ensure that you click the option just below Create Event Web Service.

You’ll need these URLs when you configure the rest of the integration.

Install and configure Splunk

Install the xMatters Alert Actions

In the Splunk UI, click the + icon to head over to the App Store:

In the App Store, search for xMatters and click the install button:

Configure the integration

Click the gear icon to navigate to the list of apps. If necessary, search for the xMatters app, and then click Setup:

Paste in the web service URL for your form, enter the username and password for your REST web services user, and then click Save:

Create an Alert

At this point, everything is configured, and now we just need to tell Splunk to use the new Alert Action. This will vary depending on your use, so this is an example. We are searching for "MY ERROR message here" in all the indexes. After entering the search, click the Save As > Alert option:

Enter a Name and Description and set the Alert Type to Real-time. Finally, click Add Actions and select xMatters. In the xMatters recipients, enter a semicolon-separated list of recipients (users or groups):

Test the integration

Triggering the alert will depend on the search criteria set up. In the example above, the text MY ERROR message here will trigger the alert:

Troubleshooting

The $SPLUNK_HOME/var/log/splunkd.log file contains information about the xMatters python script that creates the event. Either inspect this log visually, or use the Splunk web interface to search. The xMatters Alert Action uses the sendmodalert logging channel for debugging. To enable debug level:

  1. Click Settings > Server settings > Server Logging
  2. In the search box on the right side, enter sendmodalert to display this alert channel
  3. Click sendmodalert and change the logging level to DEBUG. 

The xMatters.py script will then dump debug level information to the splunkd.log file. 

Use the following search to show the latest entries:
index=_internal sourcetype=splunkd

Add sendmodalert to see information printed for the xMatters Alert Action. For example, this shows a successful event creation and shows the event ID returned from xMatters:

Download resources

Have more questions? Submit a request

8 Comments

  • 1
    Avatar
    Travis DePuy

    Huh, that is interesting. Does your user have the "admin" role in Splunk?

    Also, what do you see when you go to this page:
    http://SPLUNKHOST:PORT/en-US/manager/xmatters_alert_action/apps/local/xmatters_alert_action/setup?action=edit

     

     

  • 0
    Avatar
    Travis DePuy

    Hi Manickam, there is a config page that is accessed through the apps page. Did you see this section: https://support.xmatters.com/hc/en-us/articles/205828659#configure-the-integration? 

  • 0
    Avatar
    Manickam Annamalai


    no setup link available..

  • 0
    Avatar
    Alistair Simpson

    What is the difference between this integration and the xm_splunk_1_1 integration (https://support.xmatters.com/hc/en-us/articles/204443189-Splunk-Integration-v1-1)?

     

    alistair

  • 0
    Avatar
    Don Clark

    Hi Alistair

     

    I'll have to double check with Travis to confirm, but I believe the other version of the Splunk integration (v1.1) is two-way - though it would require the addition of an Integration Agent. 

  • 0
    Avatar
    Manickam Annamalai

    yes I was logged in as admin on the Search head cluster, and the link works for the setup screen...

  • 0
    Avatar
    Manickam Annamalai

    Travis,

    I have the Search head cluster and deployed this app from the deployment server, after the deployment, I don't see the setup enabled and just wondering how to enter the password?, I could update the local/alert_actions.conf with the URL(endpoint) and Username (xMuser), how about the passwords.conf?

  • 0
    Avatar
    Travis DePuy

    Yep, This one is 1 way and uses the "Alert Action" frame work new as of 6.3, while the other one you linked to is 2 way and uses a python script to fire to xMatters and the Integration Agent to process the callbacks from xMatters and logs them to a file. 

Please sign in to leave a comment.
Powered by Zendesk