Question

Why does this error message appear while configuring a connection from the xMatters app in Splunk to the xMatters instance?

ERROR: Delete user failed, cannot find the credential information with id : credential::xmatters_password:

Environment

• All versions of xMatters using a Splunk integration

Answer

Despite the contents of the message, this error is not actually caused by an incorrect username/password or missing credentials. It occurs due to a failed operation after the credential data is delivered to Splunk.

Resolving the Problem

To resolve this problem, follow the instructions below to create a password credential record on the Splunk instance. You may need to create a Splunk support ticket to request that port 8089 be opened on your Splunk instance before you run the commands.

To create the credential record in Splunk, you will need to be an administrator on the Splunk system and use a tool such as curl or Postman to send the following REST requests. Replace parameters such as “<username>” with the appropriate values for your instance.

NOTE: In the DELETE commands shown below, colons (:) are required before and after “xmatters_password” or “xmatters_itsi_password“, according to the Splunk REST API documentation.

Splunk Cloud

curl -k -u <username>:<password> --request DELETE https://<customer-instance>.splunkcloud.com:8089/servicesNS/nobody/xmatters_alert_action/storage/passwords/:xmatters_password:
curl -k -u <username>:<password> --request POST https://<customer-instance>.splunkcloud.com:8089/servicesNS/nobody/xmatters_alert_action/storage/passwords/ -d name=xmatters_password -d password=<new password>

Splunk ITSI

curl -k -u <username>:<password> --request DELETE https://itsi-<customer-instance>.splunkcloud.com:8089/servicesNS/nobody/xmatters_itsi/storage/passwords/:xmatters_itsi_password:
curl -k -u <username>:<password> --request POST https://itsi-<customer-instance>.splunkcloud.com:8089/servicesNS/nobody/xmatters_itsi/storage/passwords/ -d name=xmatters_itsi_password -d password=<new password>

Splunk on-premise

Use the same commands as for Splunk cloud, but replace

https://<customer-instance>.splunkcloud.com:8089/

with

https://localhost:8089/ 

After you have deleted and then recreated the password record, you should no longer encounter the error message.

Troubleshooting

• If you're working with an on-premise Splunk server and the response to the curl commands is "Unauthorized", there is probably a problem with the Splunk username/password that you're using.  

<?xml version="1.0" encoding="utf-8"?>
<response>
  <messages>
    <msg type="ERROR">Unauthorized</msg>
  </messages>
</response>

For troubleshooting purposes, try sending one of the requests using the admin username and password.  Information on managing Splunk accounts and passwords is here.

• If you are using a password with non-alphanumeric characters, try using Postman instead of Curl.  It's possible to work around this problem in Curl, but probably easier to use a tool like Postman that provides a GUI. 
• If you're working with an on-premise Splunk server and the response to the curl  commands is "Couldn't connect to server", Splunk might be using a different port.  Try port 8090.  You might need to ask Splunk support to help you find the correct port.
• If you're absolutely stuck and cannot get the HTTPS requests to work with your on-premise Splunk server, then you can log on as a Splunk admin and create the file manually.
  1. Use a text editor to create a file called password.conf with contents
     [credential::xmatters_password:]
     password = <the password>
  2. Save the file in 
     /opt/splunk/etc/apps/xmatters_alert_action/local/passwords.conf
• If your on-premise Splunk installation has multiple search heads (AKA a "distributed environment"), then the xMatters application needs to be deployed to each search head to work properly.
• If your on-premise Splunk servers are on a network that uses a proxy server to control outgoing  traffic, then you can add these settings to splunk-launch.conf located under $SPLUNK_HOME/etc :
  HTTP_PROXY = http://<proxy_server>:<port>
  HTTPS_PROXY = http:// <proxy_server>:<port>
  PROXY_RULES = <company-name>.xmatters.com
  NO_PROXY = localhost, <other dns or ip>