Submit a Request

Configuring the xMatters Agent to receive HTTPS requests

Overview

The xMatters Agent communicates with xMatters via a secure Websocket connection, but when accepting requests via its LAN interface (port 8081), the agent only accepts HTTP requests.

Some customers prefer their xAgent to receive HTTPS requests on its LAN interface, for increased security.  This article describes how to configure the agent for HTTPS.

Requirements

• In order for the xMatters Agent to accept HTTPS requests, an SSL certificate must be created for the server upon which the agent is installed. Although it's possible to use a self-signed certificate for this purpose, we will assume that the certificate has been purchased from a commercial provider such as Digicert, Verisign, etc. The certificate should be in JKS or PKCS12 format.  PKCS12 is recommended, since it is easier to ensure that the entire certificate chain is included in the resulting file.  The certificate vendor will have instructions for converting the certificate to one of these formats if necessary.
Note: the keystore file must have the same password as the certificate.  See "Troubleshooting" below for details.

• If the customer plans on using a self-signed (for example, non-commercial) certificate, then the customer should understand that all clients sending requests to the xAgent will need to be configured to accept the non-secure responses provided by the xAgent.

• Avoid storing the certificate in a folder belonging to the xAgent, since those folders may be deleted during an upgrade.

• Similarly, be sure to create a backup of the customized configuration file after you have the agent running with the new certificate.

Configuring the xAgent to receive HTTPS requests

The xAgent's configuration files can be edited only by an administrator on Windows hosts and the root user on Linux hosts. You will need to be logged on as an administrator to perform the following steps.

Windows

1. Open the Windows File Explorer and navigate to c:\program files\xa\config.

2. Create a copy of xagent-service.conf and save the copy as xagent-service.conf.bak.

3. In the Windows Search bar, search for: notepad. Right-click on the notepad icon and choose "run as administrator".

4. In Notepad, open c:\program files\xa\config\xagent-service.conf.

5. Search for wrapper.java.additional. There will be several lines beginning with this string. After the last line beginning "wrapper.java.additional", paste the following lines:
# The format used for the keystore. It can be JKS or PKCS12

6. In the new lines, change the numbers following wrapper.java.additional so they are consecutive with the existing lines. For example, if the last existing line begins with wrapper.java.additional.15, then the new lines should be numbered wrapper.java.additional.16, wrapper.java.additional.17, etc.  The numbers must be consecutive and unique (ie, no duplication of numbers and no gaps.)

7. In the new lines, change the store-type so that it matches the format of the certificate file.

8. Change the path and certificate name, store-password, and alias. (See Troubleshooting if you don't know the alias or the format.)

9. Save the file.

Linux

1. Log on as a user with root privileges.

2. cd to /opt/xmatters/xa/bin

3. copy systemd.start.sh to systemd.start.sh.bak

4. Open systemd.start.sh with your favourite text editor and replace this line:
${SPRINGBOOTAPP_JAVA} -jar /opt/xmatters/xa/bin/xa.jar with${SPRINGBOOTAPP_JAVA} -jar /opt/xmatters/xa/bin/xa.jar --server.ssl.key-store-type=JKS --server.ssl.key-store=/path/to/certificate-file.jks --server.ssl.key-store-password=changeit --server.ssl.key-alias=certificate-alias --server.ssl.enabled=true
Note that the new line should have no line breaks, and there should be a space before each "--".

5. In the new line, change the store-type so that it matches the format of the certificate file.  Valid store-types are JKS and PKCS12.

6. Change the path and certificate name, store-password, and alias. See the troubleshooting section below if you don't know the alias or the format.

7. Save the file.  Note that the file is write-protected, so you will probably be challenged by your editor when you try to save it.  In vi, you can force-save the file with [ESC]wq!

Testing

To test the xAgent's functionality with the new certificate, open a browser (from a computer other than the xAgent's server), and access the xAgent's health endpoint by going to: https://server-name.domain.com:8081/health. The expected response is a JSON object beginning with: '{"status":"UP"'. The browser should display a green padlock next to the URL.

Troubleshooting

• If the agent won't start after you have configured it, or if the console/log file displays errors, use the keytool executable in the xAgent's JRE\bin folder to verify that the certificate file is valid. The command and output should be similar to:
c:\Program Files\xa\jre\windows\jdk-11.0.4\bin>keytool.exe -list -storepass changeit -keystore "c:\path\to\test-store.jks"

Keystore type: PKCS12
Keystore provider: SUN
test, Feb 9, 2021, PrivateKeyEntry

(In the above example, "test" on the last line is the alias of the certificate.)

• If the Agent doesn't appear to be running after you issue the start command, look in the log file for errors.  The log file locations are listed here: https://help.xmatters.com/ondemand/xmodwelcome/xmattersagent/monitor-xmatters-agent.htm
If the following message appears in the log file, it indicates that the certificate password does not match the keystore password:
"java.security.UnrecoverableKeyException: Cannot recover key"
To resolve this problem, change the keystore password so that it matches the password that was used to create the SSL certificate originally.  You can change the keystore password with this command:
keytool -storepasswd -keystore /path/to/keystore
Change "/path/to/keystore" to match the path and filename of the JKS or PKCS file containing your certificate, then press Enter.  You will be prompted for the keystore's existing password, and then for the new password.  Make sure that the new password matches the one that you used when creating the original SSL certificate.  Finally, don't forget to update the "-store-password" parameter in the "systemd.start.sh" or "xagent-service.conf" file.
• On Linux, if you see a response like this after sending an HTTPS request to the xAgent, it means that the agent is not running in HTTPS mode and will accept only HTTP requests:
SSL received a record that exceeded the maximum permissible length.
• If you get a "Connection refused" response from the agent after configuring for HTTPS and restarting, check the path and filename that you provided in your configuration.  Incorrect path or filename will cause the "Connection refused" response regardless of whether your request is sent via http or https.  Use keytool to test the certificate as described above, with the exact path and filename from your configuration file.
• A useful diagnostic while troubleshooting is to use a browser as mentioned previously.  If the browser doesn't display a little green padlock before the URL, this indicates that the browser doesn't trust the SSL certificate.  It will probably display a little red triangle instead of the padlock.  Click on the triangle and you should be presented with additional information about the certificate and its flaws.  The exact format of the information varies from one operating system to another and from browser to browser.

For additional troubleshooting information, see SSL Troubleshooting for xMatters Integrations.

Note: xMatters support workers who configure an xAgent on behalf of a customer are requested to add the customer name and/or a link to the support ticket to this Enhancement Request.