SSL Troubleshooting for xMatters Integrations

This KBA is intended to help client assistance workers resolve SSL*-related problems with minimal delays and complication.  The focus of this article is on troubleshooting SSL connections between the integration agent and a management system using information and techniques garnered from past successes, but with some interpretation the information can be applied in other situations.

1. Overview

This document will make extensive use of the terms "client" and "server".  From here on, a server is a device that is connected to the internet and that is capable of responding to an HTTPS request, and a client is a device that initiates the HTTPS request. 

For example, your web browser is a client that allows you to send requests to the website of a bank or an online store.  Similarly, an Integration Agent acts as a client when it sends notification event data to an xMatters hosted instance.  Note that in this case, the client is actually transmitting data, not requesting it. The Integration Agent can also act as a server if it is configured to receive notification data from a management system such as BMC Remedy.  

So the critical distinction between a client and a server is that the client initiates the communication with the server by making a request - even if the purpose of the request is to send data to the server.

If the purpose of the request is to start an online banking session, then the client must protect the user's data by ensuring that the response is actually coming from the bank and not an impostor, and by encrypting the user's data so that it can't be stolen by a third party.  Encryption and Identity verification are performed through the use of an SSL certificate, which is received from the client as part of the response from the server.

So: an SSL certificate is a software component that is installed on a server, and that allows clients to verify the server's identity and encrypt data that is exchanged between the server and the client.

If you're new to the world of SSL certificates, there's a great introduction to the topic here; scroll down to the section headed "How Does the SSL Certificate Create a Secure Connection?"

1.1 Commercial vs. Self-Signed Certificates

Commercial SSL certificates (purchased from a recognized vendor such as Digicert, Verisign, etc) allow any client to verify that the information it is receiving is trustworthy, in the sense that the information is being provided by the proper server.  Because these commercial certificates can be algorithmically linked to a trusted certificate vendor, and no client configuration is required, they are essential to businesses providing online banking, credit card payments, and exchange of confidential information.

However, there is a disadvantage to these commercial certificates: they cost money.

Many of the advantages of a commercial certificate can be obtained through use of the self-generated (aka "self-signed") certificates.  These certificates provide most of the benefits of a commercial certificate and they are free, but there is a disadvantage: a client cannot trust a self-signed certificate unless a copy of it is manually added to the client's trust store.  This is obviously unappealing in an online banking situation, but might be perfectly practical when transferring data securely between a management system and an IA.

Plenty of information is available pertaining to setting up a certificate on a server, but relatively little that applies to clients.  Adding a self-signed certificate to the IA's trust store is one of the topics of this KBA.

1.2 Certificate Chains

When the owner of a server buys an SSL certificate from a vendor such as Verisign, they will receive several certificate files.  There will be a copy of the vendor's certificate, a certificate that is unique and specific to the purchaser, and usually an "intermediate" certificate.  The vendor's certificate does not have to be directly trusted by the client (eg, the browser that you are using to do your online banking).  The concept of certificate chaining allows the client to trust a certificate even if it is not in the client's trust store.  

The client's Trust Store will contain a relatively small number of "Root" certificates that are considered trustworthy by the provider of the user's browser, JRE, or operating system.

A certificate chain is a set of certificates that links a server's certificate to one of the "root" certificates that the client trusts.  When examining the certificate data received from a server, a client will first test the server's certificate to see if it matches one of the certs in the client's trust store.  If it doesn't, then the client will examine the first intermediate certificate in the data.  This intermediate certificate is digitally linked to the server's own certificate, so the client tests the link between these two to ensure that it is valid.  The client verifies that each intermediate certificate in the chain is authentically linked to the next one.  When it gets to the end of the chain, it expects to find a certificate that matches one of the Root  certificates in its own trust store.  Assuming that is the case, then the server's certificate is considered trustworthy and the client continues the exchange.

A good way to view a certificate chain is to use a browser to access the server (eg, https://www.xmatters.com).  You can use the browser to display the "certification path" if desired, but simply browsing to the URL and looking at the browser's green padlock icon will tell you whether the certificate is valid.

In the Google Chrome browser's "Certification Path" screen you can view the certificate chain.  At the top is the issuer's Root certificate, and at the bottom is the certificate belonging to the server.  Each certificate is verifiably linked to the one above it in the chain.

1.3 Certificates on the Server

Typically, when an SSL certificate is purchased (eg, from Verisign,) the purchaser will receive a unique certificate that contains data about the server and the organization that owns it.  The purchaser will generally also receive one or more "intermediate" certificates and a Root certificate.  The purchaser will install the root, intermediate, and server certificates in the keystore file on the server.  

Once the certificates are properly installed on the server, the server is able to accept requests starting "https://".  When an HTTPS request comes in (eg, from a browser or an Integration Agent), the server will provide a response that includes the server, intermediate, and root certificate. 

If the certificates are not installed correctly in the server's keystore, then the requester will see a warning that the chain is not valid, and the site is not to be trusted.  This will also be the case if the server certificate has expired or has been revoked.

You can inspect the certificate's validity dates and revocation status by using a browser (as described in the previous section).

1.4 Client Authentication Certificates

This article is mainly concerned with Server certificates (ie, the certificate that is presented by the Server when it responds to a request from a Client.)  However, it is possible for a Server to require that the Client provides a certificate with the request.  If this is the case, the Server will not allow the connection to proceed if the Client does not provide its own SSL certificate.

Client Authentication requires two things from the client:

• The client must have a valid SSL certificate to present when initiating a request.
• The client must include specific software components that allow it to provide the certificate when making a request.  

The xMatters Integration Agent does not have the software components that are required to perform Client Authentication.  Therefore, the IA cannot create connections to Servers that require Client Authentication Certificates.

 

2. Adding a Certificate to the IA's Trust Store

For SSL communication to work, the JRE may need to possess a local copy of the server's public certificate. This will be necessary if

• the server's certificate is self-signed, or
• there is a problem with the server's commercial certificate.

The client's copy is kept in a "trust store", which by default is a file called cacerts in the <IAHOME>\jre\lib\security folder. 

To add a certificate to the JRE's trust store, make a backup copy of the cacerts file, then open a command window in the Integration Agent's jre\bin folder and run the following command:

keytool -importcert -keystore ../lib/security/cacerts -storepass changeit -file /temp/somecert.cer -alias somecert

• <IAHOME>/jre/lib/security/cacerts is the JRE's default trust store. You can use a different trust store, but that will require specifying the path in the IA's wrapper.conf. It is very unlikely that you'll need to do this.
• "changeit" is the default password for any JRE's trust store.
• -file is followed by the path to the certificate that you will be adding to the trust store. The certificate should be in either DER (binary) format, or else X.509 Base-64 encoded text format.  Certificates exported from Chrome, for example, will be DER by default.  JRE 8 will also trust PKCS12-formatted certificates in "compatibility" mode.
• -alias allows you to specify an alias for the certificate, which makes it more convenient to list and manipulate the certificates in the store.

You can view certificates in the store with the keytool -list command. To see only a single certificate, use the -alias argument. For more detail, use -v:

keytool -list -keystore ../lib/security/cacerts -storepass changeit -alias somecert -v

Where to get a copy of the certificate? The customer will probably be able to provide one, but you can also export a copy from a browser. Simply browse to the server's URL and then use the browser's certificate export tool. Browser-specific instructions are available via Google search.

3. Configuring Integrations For SSL

Integrations that are designed to allow SSL communication with the management system may include configurable parameters for the trust store's location and filename, password and so on.  Refer to the integration document for an example.

This capability is provided by XMIO.js (used by newer integrations.) Older integrations may instead use wsutil.js, which is part of the integration agent utilities package.  If this isn't made clear by the integration document, search the integration's script files for references to XMIO.js or wsutil.js, and follow the path to determine which library is providing the functionality.  XMIO.js uses the JRE's global settings, including the cacerts trust store mentioned above.  Wsutil.js uses a different trust store.  See the comments in wsutil.js for details.

4. Diagnosing SSL Problems

Diagnosing SSL connection failures is difficult for several reasons:

• The information that is logged by the Java libraries tends to be vague and not very helpful.
• SSL connections can fail for a variety of reasons, some of which are difficult to identify
• the server's certificate may have expired, been revoked, may not be correctly installed, may not be compatible with the TLSv1.2 protocol, or its "Valid from" date may be in the future.
• SSL communication is not well understood by many IT workers.

In this chapter we will describe some troubleshooting techniques and some tools that we have found to be useful.

If the terminology in this chapter is unfamiliar, read the Background Information chapter near the end of this document.

4.1 Messages in the IA Log

The following message in an IA log indicates that you have encountered an SSL certificate problem:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The "certification path" in the message is a reference to the certificate chain mentioned earlier on.  The "requested target" is the organization name in the certificate, which must match the URL to which the request was sent.

4.2 Ensure that the Trust Store has a Valid Certificate for the Server

If the certificate is a commercial one, and the IA is writing SSL errors in its log file, then you can use a browser to get a "second opinion" on the certificate's validity.  If the browser and the IA agree that the certificate is not valid, then you can work with the customer to have a valid certificate installed on their server.  

 

Even if the browser shows the certificate to be valid, it is still possible that the IA's JRE will not trust it. For example, there was a period during which GoDaddy's root certificate was not included in the Oracle JRE's trust store, and therefore certs issued by GoDaddy were not recognized by Java applications. Keep in mind these differences between JREs and browsers:

• They are produced by different development teams.
• They use different trust stores.
• They don't get updated at the same time (browsers tend to get updated automatically, whereas JREs often go unchanged for years.)

Given these differences, it's not surprising that the IA's JRE and your browser might disagree on whether a certificate is trustworthy. If in doubt, install a copy of the server certificate in the IA's trust store.

If the server's SSL certificate is self-signed (ie, not a commercial certificate,) then a copy of it must be installed in the IA's trust store. 

In either case, ensure that:

• The certificate has not expired.
• It has not been revoked.
• It has the correct information in its Certificate Name ("CN=") field. The Certificate Name information must match the URL to which the IA is sending its requests.  Note that in Java 8 and higher, it is also necessary for the certificate to include a Server Alternate Name (SAN) that matches the URL to which the IA is sending its request.

Even if the server certificate is not self-signed, it might need to be installed in the IA's trust store (eg, if the certificate was not properly installed on the server.)  

If the certificate is self-signed, and you want to use the browser for SSL troubleshooting, then you can import the certificate into the browser's trust store.  A Google search will get you directions specific to your browser (IE, Firefox, Chrome, etc) and operating system.  If the certificate is not trusted by the browser even after being imported, then the certificate probably will not work with the IA either.

4.3 Inspect The Certificate with Keytool

After importing a certificate, you can inspect it to determine whether it has obvious flaws (and to verify that it was imported correctly.)  To inspect a certificate in the IA's trust store, use a utility such as keytool.exe.  Keytool is included with the Integration Agent and can be found in <IAHome>\jre\bin.  In this way you will be able to determine who issued the certificate and who it was issued to, as well as the certificate's expiry and issuance dates.

Considerations when using keytool:

• Use the keytool executable that was shipped with the integration agent.  This is important because the JRE may not be able to trust certificates in a trust store that was copied from a JRE of a different version.
• To inspect the certificate, use the following keytool command:

keytool -list -v -keystore <storeName> -alias <theAlias>

where <storeName> is the path to the trust store (eg, ..\lib\security\cacerts) and <theAlias> is the alias that was used when the certificate was added. (If you don't know the alias, omit the "-alias" parameter from the command.)

You will probably want to direct the output to a text file so you can examine it with an editor, since the keystore may contain dozens of certificates.

4.4 What to look for when inspecting a certificate

 The first few lines will contain most of the information that you need.  For example, looking at the xMatters.com certificate:

C:\xmatters\integrationagent-5.1.8\jre\bin>keytool -list -v -storepass changeit -keystore /temp/cacerts -alias xmatters
Alias name: xmatters
Creation date: Feb 27, 2017
Entry type: trustedCertEntry

Owner: CN=*.xmatters.com, O="xMatters, inc.", L=San Ramon, ST=CA, C=US
Issuer: CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US 

Valid from: Thu Feb 18 16:00:00 PST 2016 until: Fri Feb 22 04:00:00 PST 2019

Also pay attention to the "SubjectAlternativeName" section later on in the report:

SubjectAlternativeName [
DNSName: *.xmatters.com
DNSName: *.hosted.xmatters.com
DNSName: *.api.xmatters.com
DNSName: *.na1.xmatters.com
DNSName: *.na2.xmatters.com
DNSName: *.na3.xmatters.com
DNSName: *.na4.xmatters.com
DNSName: *.na5.xmatters.com
DNSName: *.cs1.xmatters.com
DNSName: *.eu1.xmatters.com
DNSName: xmatters.com
]

This information tells us that the certificate is valid for any URL that ends with ".xmatters.com".  It also tells when the certificate will expire, and who issued it.  

The connection will not succeed if the URL is "https://35.201.109.29", unless the SubjectAlternativeName field includes an entry "ip:35.201.109.29", even though www.xmatters.com resolves to this address. Nor will the connection succeed if the JRE cannot resolve the name www.xmatters.com (if this is the case, add an entry in the client's hosts file,) or if the JRE cannot reach the address because of interference from a firewall or proxy server.

Note that this is a so-called "wildcard" certificate, which can be installed on a number of servers. More typically, a certificate will be valid for only one server.  

Note that the server's IP address must be present in the Subject Alternative Name (SAN) field, if the https request uses the server's IP address.  If the IP address is present in the Certificate Name (CN) field but not the SAN field, then the connection will fail with "java.security.cert.CertificateException: No subject alternative names present".

If the "Valid Until" date has expired, the connection will not succeed. Similarly, if there is a "Revocation Time:", the certificate will not be accepted. Certificate issuers occasionally revoke certificates if there is a problem with them - for example, if a vulnerability such as heartbleed is discovered. In such cases, the server administrator should be able to get a replacement certificate from the issuer.

4.5 Testing the Connection

4.5.1 Using Ssltest

If the certificate in the trust store is valid, and the integration is configured to use the correct trust store, then the connection should work.  If it appears not to be working, then ssltest will let you test the connection using the IA's JRE, but without having to use the IA.  This is desirable when the IA is being used in production, and also lets you avoid the clutter of the IA's console and/or logs.  

To do this:

• use ssltest.class, which is attached to this KBA.  Ssltest is a very simple Java utility which will test the connection to the server that you specify, using the same JRE as the integration agent, and reports success or failure.
  1. download ssltest.class and copy it to <IAHOME>\jre\bin (to ensure that the test uses the same JRE as the integration agent)
  2. open a command window in <IAHOME>\jre\bin
  3. invoke ssltest with the following command:

java ssltest <url>

• for example,

java ssltest https://www.xmatters.com

• if the certificate is not in the IA's default trust store (<IAHome>/jre/lib/security/cacerts), use the following syntax to specify the trust store and password:

java -Djavax.net.ssl.trustStore=</path/to/cacerts> -Djavax.net.ssl.trustStorePassword=<password> ssltest <url>

• for example,

java -Djavax.net.ssl.trustStore=../lib/security/cacerts -Djavax.net.ssl.trustStorePassword=changeit ssltest https://www.xmatters.com

• The URL will probably be reported in the error message in the IA log.  If not, then look for the URL in the integration script file.  It is likely that the integration is using the URL to connect to the management system's web services.  If this is the case, use the WSDL to test the connection.  Ask the management system administrator for the URL that provides access to the WSDL.

• The expected output is:

C:\xMatters\integrationagent-5.2.2\jre\bin>java ssltest https://www.xmatters.com
[INFO]  Received host address https://www.xmatters.com
[INFO]  Setting connection timeout to 5 second(s).
[INFO]  Trying to connect to https://www.xmatters.com
[INFO]  Great! It worked. 

• If the certificate has expired, the result will be

[INFO] Could not connect to the host address https://expired.badssl.com
[INFO]  The error is: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed.

• If the certificate specifies a key that is not secure enough to comply with TLSv1.2, the result will be

[INFO] Could not connect to the host address https://dh512.badssl.com
[INFO] The error is: DHPublicKey does not comply to algorithm constraints
[INFO] Here are the details:
[SEVERE] DHPublicKey does not comply to algorithm constraints. 

There is a variety of bad certificate types accessible at bad-ssl.com.  By testing your troubleshooting tool against these certificates, you can reproduce the error that you are experiencing and thereby identify the cause of the error you are diagnosing. 

4.5.2 Using Analyze-ssl.pl

This is another excellent tool for SSL connection diagnosis.  Unfortunately, it has a couple of disadvantages:

• It requires a perl interpreter, which does not come pre-installed on most servers.  Some clients might balk at installing additional software on their IA server.  
• It doesn't use the same Java libraries or the same trust store as the IA, so it can't be relied upon to reproduce errors encountered by the IA every time.

Analyze-ssl.pl is compatible with ActiveState perl 5.24.1 and higher, which is available for Windows, OS X and Linux.  You can download ActiveState perl from https://www.perl.org/get.html

If you need to diagnose an SSL connection and have access to a suitable version of perl, then this tool is well worth the trouble.  Its output includes excellent information that simply isn't available from Java-based tools.

You can download the script from https://github.com/noxxi/p5-ssl-tools/blob/master/analyze-ssl.pl or you can use the copy that is attached to this document.

Syntax: perl analyze-ssl.pl <with no arguments> - display the help screeen

perl analyze-ssl.pl (options) <address> -analyze the SSL/TLS connection to the address

Example: perl d:\tools\analyze-ssl.pl --show-chain xmsupport.hosted.xmatters.com

Output:

-- xmsupport.hosted.xmatters.com port 443
* maximum SSL version : TLSv1_2 (SSLv23)
* supported SSL versions with handshake used and preferred cipher(s):
* handshake protocols ciphers
* SSLv23 TLSv1_2 ECDHE-RSA-AES128-GCM-SHA256
* TLSv1_2 TLSv1_2 ECDHE-RSA-AES128-GCM-SHA256
* TLSv1_1 TLSv1_1 ECDHE-RSA-AES128-SHA
* TLSv1 TLSv1 ECDHE-RSA-AES128-SHA
* cipher order by : server
* SNI supported : ok
* certificate verified : ok
* chain on 65.151.10.10
* [0/0] bits=2048, ocsp_uri=http://ocsp.digicert.com, /C=US/ST=CA/L=San Ramon/O=xMatters, inc./CN=*.hosted.xmatters.com SAN=DNS:*.hosted.xmatters.com,DNS:hosted.xmatters.com
* [1/1] bits=2048, ocsp_uri=http://ocsp.digicert.com, /C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
* [-/2] bits=2048, ocsp_uri=, /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
* OCSP stapling : no stapled response
* OCSP status : good

This option, which analyzes the certificate chain, is one of the most useful features of the analyze-ssl tool (particularly because this functionality doesn't seem available at all in the Java world.)  There are other options, plus a verbose mode.  See the help mode for details, by invoking analyze-ssl.pl with no arguments.

If the client won't let you install this tool on the IA server, it works just as well from a laptop.  Ideally, you will use a wired connection to the same switch as the IA is using, so that you won't get false results due to proxy servers, etc.

Keep in mind that this tool uses a different trust store and different algorithms from the IA's JRE.  So, as with a browser, there is a chance that you will get different results from this tool than you get from the IA or ssltest.

4.5.3 Using ssllabs.com

If the server is internet-accessible, then you can paste its URL into www.ssllabs.com to get a comprehensive report on its SSL certificate.  Despite the excellent information available from this site, there are a couple of deficiencies:

• It only works with publicly-accessible servers, so if you are trying to diagnose a certificate belonging to a server on a LAN or a proxy server, then ssllabs will probably not be able to help you.
• As with alalyze-ssl.pl and the browser, ssllabs.com uses a different certificate store and different algorithms from the IA's JRE.  So there is a chance that you will get different results from this tool than you get from the ssltest or from the IA itself.

4.5.4 Using curl

curl is available for Linux and Windows, and it is a great tool for testing SSL certificates.  It provides detailed information about the certificate, and you can provide arguments to specify the TLS (or SSL) level, the proxy parameters (if applicable), and a great variety of other parameters.  User "curl -h" to see them.  The help is terse, but more details and examples are available online. 

The basic command is 

curl -v https://www.xmatters.com

Note that syntax of some commands is different for Windows than for Linux.

You can redirect the actual page content to a file, so that your console shows only the connection data.  For example, 

curl -v -o c:\temp\curl-output.txt https://www.xmatters.com

Disadvantages to using curl are:

• It isn't a Java program, so it can provide false positives.  In other words, just because a connection succeeds with curl, it won't necessarily succeed for the IA.
• It isn't installed by default on Windows servers (or on some Linux systems,) and so a customer may not allow it to be installed on their server.

However, curl provides great diagnostic details that are not available from Java (and therefore they aren't available from the IA itself.)  So if you are looking for detailed troubleshooting information, curl provides some of the best that you will find.

Tools such as curl (or a browser) are also useful in that they can demonstrate to a customer that the IA is not the source of connection problems.  If you can demonstrate to the customer that connections fail with curl as well with the IA, then the IA itself is less likely to be blamed for the problem.

Note that recent versions of curl for Windows do not seem to display certificate information.  The message to watch for is 

schannel: verifyhost setting prevents Schannel from comparing the supplied target name with the subject names in server certificates.

This was encountered with curl for Windows 7.5.5, so use an earlier version instead.  curl 7.4.0 is recommended. 

5 Advanced Diagnostic Procedures

If the above tools fail to help you resolve the problem, then there are still a few things you can do.

5.1 Re-checking the Basics:

• ensure that the certificate is valid (and hasn't expired)
• ensure that you are testing with a URL that matches the Certificate Name shown in the trust store
• ensure that the integration agent server can resolve the management server's network name, exactly as it appears in the Certificate Name
• ensure that there is no interference from a  firewall or a proxy server between the integration agent server and the management system

5.2 Using Ssltest in Debug Mode

If you pass the debug=all flag to the JRE while invoking ssltool, the output will be verbose but potentially very helpful in diagnosing the connection failure.  By comparing the tool's debug-level output with the attached file, you will be able to determine exactly which step failed in the SSL exchange.

Open the attached ssltest.debug.log file with a text editor to see examples of the following information from a successful exchange:

• truststore name and path
• list of certificates found in the truststore (abbreviated in this example)
• SSL handshake (initial "write" from client and "read" response from server "vic-vw-remedy8.invoqsystems.com")
• analysis of the certificate chain sent by the server
• "Found trusted certificate" message indicating successful match of the server certificate with one found in the truststore
• request for data, using the secure connection
• decrypted information received from the server
• "close" message sent by client
• client closes the socket connection

Compare the information in this file with the output from your own session.  

Use the following syntax to run ssltest in debug mode:

<IAHOME>\jre\bin\java.exe -Djavax.net.debug=all ssltest <URL>

To simplify inspection of the output, redirect it to a text file.  For example:

<IAHOME>\jre\bin\java.exe -Djavax.net.debug=all ssltest https://www.xmatters.com/ > %temp%\ssltest.txt

Hint: when opening the output file in a text editor, search for "***".  This string is used as a marker to indicate milestones in the establishment of an SSL connection.

Here are some notes on interpreting the verbose output:

• The messages do not discriminate between TLS v. 1.0, TLS v. 1.1 and TLS v. 1.2, so don't spend time investigating the possibility that TLS 1.0 is being used instead of TLS 1.2.  Assuming that you are using IA 5.1.8 (which ships with JRE 8) or later, TLS 1.2 is used by default.  
• Don't be deceived by the messages saying "adding as trusted cert:".  Those messages indicate that the JRE is reading its trust store file and making a list of candidate certificates to be potentially matched against the ones submitted by the server.  The messages don't indicate that a server's certificate will automatically be trusted just because it has the same "CN=" name.
• The "*** Certificate chain" messages are worth paying attention to.  Typically the certificate chain will be an array of one or more certificates which have been submitted by the server, and the client will attempt to match those certificates to the ones in its trust store.  If the chain has multiple certificates, then the first one (labeled "chain [0]") will be the server's own (eg, "CN=*.xmatters.com") and the rest ("chain [1]", etc) will be a "root" from the certificate authority (the vendor) who provided the certificates.
• The "certificate chain" messages should be followed by  "***
  Found trusted certificate: "
• If the "certificate chain" messages are followed instead by messages like these, then there is a problem with the certificates sent by the server:
  "%% Invalidated: [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA]"
  "TLSv1 ALERT: fatal, description = certificate_unknown "
  The problem is likely to be one of:
• The "root" certificate from the server is not recognized as a Certificate Authority.  The root cert should include a record saying "CA:true", and should match one of the certs in the IA's trust store.
• The IA's JRE could not create a valid link from the server certificate to the root certificate. For example, if the certificate chain is intended to provide an intermediate certificate as well as a root and a server certificate (which is often the case,) but the intermediate certificate is absent, then the X509 Trust Manager will not be able to validate the server certificate and the connection will fail.  In other words, the certificate was not properly installed on the server.  A browser should be able to confirm this diagnosis.
• The server's certificate was not added correctly to the IA's trust store.  One likely cause is that the certificate was not in a form that the IA's JRE could accept (using keytool - list should tell you if this is the case.)  
  Another possibility is that the server's entire certificate chain was installed in the IA's trust store.  It's a mistake to install the server's intermediate and root certificate in the IA's trust store.  If you install the server's certificate alone (whether or not it is self-signed,) the IA will trust it - assuming that the certificate is in a format that is acceptable to the IA's JRE.  Again, keystore -list will tell you this.

If these notes don't help you to resolve the problem, then take a look at the attached Oracle "Debugging SSL/TLS Connections" document for further information about the certificate validation and handshaking process. 

There is also excellent information about each phase of an SSL/TLS session in "Traffic Analysis of an SSL/TLS Session"

5.3 Compiling Ssltest on the Server

If the client will not agree to installing the ssltest binary file on the server, they may agree to allowing the ssltest tool to be compiled from the source code attached to this KBA.  Provide the file ssltest.java to the client - it is a very simple piece of code that can be inspected with a text editor.  Once it has been inspected, the client can compile it with the following command (assuming that the Java JDK is installed):

javac ssltest.java

This will create a new copy of the ssltest.class file which can be used as documented above.

6. Background Information

All these terms can be so confusing!  Troubleshooting is hard enough without a bunch of jargon getting in the way.  Here are explanations of a few of the terms used in this KBA.

SSL ("secure sockets layer") is a mechanism that allows HTTP requests and responses to be transmitted securely.  HTTP plus SSL is where HTTPS comes from.  SSL is also used in other forms of communication, such as SSH (which allows people to log on to and control remote servers,) and SCP (which allows files to be transferred securely.)

Two major parts of SSL communications are encryption and verification. Encryption ensures that a third party who intercepts a request or response will not be able to understand the content. Verification ensures that a client can trust the response that they receive from a server, by verifying the responder's identity. That is what SSL certificates are for.

*TLS is a newer version of the Secure Sockets Layer specification. In this KBA I've avoided referring to TLS because the term "SSL" is woven all through the terminology (e.g., no-one uses the term "TLS certificates"), and it wouldn't have added clarity.  

A keystore is just a file or other storage medium that contains SSL keys and/or certificates. The keys and certificates have to be added via software such as the Java keytool.  

Wait a minute... so far we've been talking about putting "certificates", not "keys", into keystores. So what's this "key" business? Answer: an SSL key (actually, a key pair) is a unique two-part digital document that is required to create an SSL certificate (either commercial or self-signed.) The SSL key pair consists of a public key, which is included in the certificate, and a private key, which is kept on the server and is never shared. 

For you keeners, there is a good overview of SSL/TLS encryption here: https://www.digicert.com/ssl.htm

A server's keystore holds its private key and its SSL certificate. When the client receives a response from the server, it uses the certificate information in the response to verify the server's authenticity. The client also uses the public key from the certificate to encrypt its next request. The server uses its private key to decrypt the request.

A trust store is also a file that contains certificates. The main difference between a keystore and a trust store is that a trust store is used by a client (i.e., the device that initiates a request to a server), whereas a keystore is used by a server. Another difference is that a keystore contains the server's private key and its certificate, whereas a trust store contains only certificates. When the client receives an HTTPS response, it compares the certificate information provided by the server with the certificates in its trust store, to see if there is a match. If none of the certificates match, then the response will not be trusted.  

Different devices and systems store their trusted certificates in different ways. The Windows OS stores its trusted certificates in the Registry. Some browsers such as Chrome and Internet Explorer use the trusted certificates provided by the OS. Other browsers such as Firefox and Opera keep their own separate trust store, in a file. And the Java JRE also keeps its trusted certificates in a file, as described above.

Since the JRE can operate as a server or a client, it needs a keystore and also a trust store. By default, the JRE's keystore and trust store are the same file.

In the JRE's default trust store ("cacerts"), you will see a number of certificates that were shipped with the JRE. These certificates are in the trust store because they are issued by Certificate Authorities; generally, these are the companies that sell SSL certificates.

If a server sends a certificate that was supplied by a commercial signing authority, but the certificate was not installed correctly on the management system, then it may not be recognized by the JRE. In other words, it will be treated like a self-signed certificate. And, like a self-signed certificate, the IA's JRE will trust an improperly-installed server certificate, if it is added to the IA's trust store.

7. SSL Inspection Proxy Servers

You may encounter difficulties connecting to xMatters On Demand through an SSL inspection proxy server.  A likely cause is that the proxy server removes the certificate to inspect the traffic and then puts a new certificate on the data to deliver it to the Integration Agent.  The agent will not accept the data if the proxy server's certificate is not trusted (eg, if the certificate is self-signed.)

One solution is to switch off SSL inspection on the proxy for that traffic.  If that's not possible, then adding the proxy server's certificate to the IA's trust store (as described in section 2) solves the problem.

(Thanks to Adam Watson for contributing this field-tested information.)

8. Further Reading

This is an excellent resource for SSL/TLS troubleshooting: https://maulwuff.de/research/ssl-debugging.html

If you want to understand SSL cryptography better, here's a great starting point:
https://www.digicert.com/ssl-cryptography.htm

If you've enabled debug logging with an SSL/TLS-enabled application, and you want to know what all those cryptic messages mean, check out this article: Traffic Analysis of an SSL/TLS Session

xMatters Reference
JDN-4346 Originally created by Jeremy Brown