This KBA is intended to help client assistance workers resolve SSL-related problems with minimal delays and complication. The focus of this article is on troubleshooting SSL connections between the integration agent and a management system using information and techniques garnered from past successes, but with some interpretation the information can be applied in other situations.
There's plenty of information available pertaining to setting up certificates on a server, but relatively little that applies to clients. Here's a quick overview of why a client (eg, the IA) needs SSL certificates when interacting with a server (eg, a management system such as Remedy.)
- When a client (such as the IA) sends an HTTPS request to a server, the client expects to receive information about the server's SSL certificates in the response.
- Unless the IA has a way of authenticating the server's certificate, it will not trust the server.
- Therefore, the IA has a file called a "trust store", which contains all of the certificates that it knows can be trusted. This includes certificates from Digicert, GoDaddy, Verisign and all of the other commercial vendors. The trust store is included with the Java Runtime Environment that ships with the IA.
- The IA compares the certificate from the server with the certificates in its trust store to see if there is a match. If there is, then the IA continues the dialog with the server and receives the data that it initially requested.
1.1 Commercial vs. Self-Signed Certificates
Commercial SSL certificates (purchased from a recognized vendor such as Digisign, Verisign, etc) allow any client to verify that the information it is receiving is trustworthy, in the sense that the information is being provided by the proper server. Because these commercial certificates can be algorithmically linked to a trusted certificate vendor, and no client configuration is required, they are essential to businesses providing online banking, credit card payments, and exchange of confidential information.
However, there is a disadvantage to these commercial certificates: they cost money.
Many of the advantages of a commercial certificate can be obtained through use of the self-generated (aka "self-signed") certificates. These certificates provide most of the benefits of a commercial certificate and they are free, but there is a disadvantage: a client cannot trust a self-signed certificate unless a copy of it is manually added to the client's trust store. This is obviously unappealing in an online banking situation, but might be perfectly practical when transferring data securely between a management system and an IA.
Adding a self-signed certificate to the IA's trust store is one of the topics of this KBA.
1.2 Certificate Chains
A certificate chain is a set of certificates that links a commercial certificate to the vendor's "root" certificate. Typically, when an SSL certificate is purchased (eg, from Verisign,) the purchaser will receive a "root" certificate and one or more "intermediate" certificates, in addition to the actual server certificate that was requested. The purchaser will install the root, intermediate, and server certificates in the keystore file on the server.
When the server receives an HTTPS request, it will provide a response that includes the server certificate (which includes information about the server itself, and the issuer - Verisign, in this example) plus the root certificate. The root certificate is expected to match one of the certificates in the client's own trust store.
If the certificates are not installed correctly in the server's keystore, then the requester will see a warning that the chain is not valid, and the site is not to be trusted. This will also be the case if the server certificate has expired or has been revoked.
A good way to test the validity of the certificate chain is to use a browser to access the server (eg, https://xmatters.com). You can use the browser to display the "certification path" if desired, but simply browsing to the URL and looking at the browser's green padlock icon will tell you whether the certificate is valid.
2. Adding a Certificate to the IA's Trust Store
For SSL communication to work, the JRE may need to possess a local copy of the server's public certificate. This will be necessary if
- the server certificate is self-signed, or
- the server certificate was not correctly installed.
The client's copy is kept in a "trust store", which by default is a file called cacerts in the <IAHOME>\jre\lib\security folder.
To add a certificate to the JRE's trust store, make a backup copy of the cacerts file, then open a command window in the IA's jre\bin folder and use the following command:
keytool -importcert -keystore ../lib/security/cacerts -storepass changeit -file /temp/somecert.cer -alias somecert
- ../lib/security/cacerts is the JRE's default trust store. You can use a different trust store, but that will require specifying the path in the IA's wrapper.conf. It is very unlikely that you'll need to do this.
- "changeit" is the default password for any JRE's trust store
- -file is followed by the path to the certificate that you will be adding to the trust store. The certificate should be in either DER (binary) format, or else X.509 Base-64 encoded text format. By default, JRE 8 will also trust PKCS12-formatted certificates in "compatibility" mode.
- -alias allows you to specify an alias for the certificate, which makes it more convenient to list and manipulate the certificates in the store.
You can view certificates in the store with the keytool -list command. To see only a single certificate, use the -alias argument. For more detail, use -v:
keytool -list -keystore../lib/security/cacerts -storepass changeit -alias somecert -v
Where to get a copy of the certificate? The client will probably be able to provide one, but you can also export a copy from a browser. Simply browse to the server's URL and then use the browser's certificate export tool. Browser-specific instructions are available via Google search.
3. Configuring Integrations For SSL
Integrations that are designed to allow SSL communication with the management system will include configurable parameters for the trust store's location and filename, password and so on. Refer to the integration document for an example.
This capability is provided by wsutil.js, which is part of the integration agent utilities package. Search your integration's script files for references to wsutil.js, and follow the path to ensure that you are editing the correct copy of this file.
4. Diagnosing SSL Problems
Diagnosing SSL connection failures is difficult for several reasons:
- The information that is logged by the Java libraries tends to be vague and not very helpful
- SSL connections can fail for a variety of reasons, some of which are difficult to identify
- the server's certificate may have expired, been revoked, may not be correctly installed, may not be compatible with the TLSv1.2 protocol, or its "Valid from" date may be in the future.
- SSL communication is not well understood by many IT workers
In this chapter we will describe some troubleshooting techniques and some tools that we have found to be useful.
If the terminology in this chapter is unfamiliar, read the Background Information chapter near the end of this document.
4.1 Messages in the IA Log
The following message in an IA log indicates that you have encountered an SSL certificate problem:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
The "certification path" in the message is a reference to the certificate chain mentioned earlier on. The "requested target" is the organization name in the certificate, which must match the URL to which the request was sent.
4.2 Ensure that the Trust Store has a Valid Certificate for the Server
If the server's SSL certificate is self-signed, then a copy of it must be installed in the IA's trust store. Ensure that the certificate has not expired, has not been revoked, and has the correct information in its Certificate Name ("CN=") field. The Certificate Name information must match the URL to which the IA is sending its requests.
Even if the server certificate is not self-signed, it might need to be installed in the IA's trust store. The certificate might not be properly installed on the server. You can use a browser to perform a quick check - if the browser doesn't display a green padlock icon, then the server's certificate will not be accepted by the IA.
Even if the browser shows the certificate to be valid, it is still possible that the IA's JRE will not trust it. For example, there was a period during which GoDaddy's root certificate was not included in the Java trust store, and therefore certs issued by GoDaddy were not recognized by Java applications. Keep in mind that the JRE and the browser were produced by different development teams, and that the JRE and the browser do not use the same trust store. If in doubt, install a copy of the server certificate in the IA's trust store (<IAHOME>\lib\security\cacerts by default).
If the certificate is self-signed, then you can import it into the browser's trust store. A Google search will get you directions specific to your browser (IE, Firefox, Chrome, etc) and operating system. The purpose of doing this is to verify whether the problem lies in the certificate itself, or in the way the certificate has been imported into the IA's trust store. If the certificate is not trusted by the browser even after being imported, then it will not work with the IA either.
4.3 Inspect The Certificate with Keytool
After importing a certificate, you can inspect it to determine whether it has obvious flaws (and to verify that it was imported correctly.) To inspect a certificate in the IA's trust store, use a utility such as keytool.exe. Keytool is included with the Integration Agent and can be found in <IAHome>\jre\bin. In this way you will be able to determine who issued the certificate and who it was issued to, as well as the certificate's expiry and issuance dates.
Considerations when using keytool:
- Use the keytool executable that was shipped with the integration agent. This is important because the JRE may not be able to read certificates that were added with a different version of keytool.
- to inspect the certificate, use the following keytool command:
keytool -list -v -keystore <storeName> -alias <theAlias>
- <storeName> is the path to the trust store (eg, ..\lib\security\cacerts) and
- <theAlias> is the alias that was used when the certificate was added. (If you don't know the alias, omit the "-alias" parameter from the command.)
You will probably want to direct the output to a text file so you can examine it with an editor, since the keystore may contain dozens of certificates.
4.4 What to look for when inspecting a certificate
The first few lines will contain most of the information that you need. For example, looking at the xMatters.com certificate:
C:\xmatters\integrationagent-5.1.8\jre\bin>keytool -list -v -storepass changeit -keystore /temp/cacerts -alias xmatters
Alias name: xmatters
Creation date: Feb 27, 2017
Entry type: trustedCertEntry
Owner: CN=*.xmatters.com, O="xMatters, inc.", L=San Ramon, ST=CA, C=US
Issuer: CN=DigiCert SHA2 Secure Server CA, O=DigiCert Inc, C=US
Valid from: Thu Feb 18 16:00:00 PST 2016 until: Fri Feb 22 04:00:00 PST 2019
This information tells us that the certificate is valid for any URL that ends with ".xmatters.com", when the certificate will expire, and who issued it. Note that this certificate would not be valid for any address ending ".hosted.xmatters.com", so a separate "*.hosted.xmatters.com" certificate is required for xMatters hosted instances.
The connection will not succeed if the URL is "https://220.127.116.11", even though www.xmatters.com resolves to this address. Nor will the connection succeed if the JRE cannot resolve the name www.xmatters.com (if this is the case, add an entry in the OS's hosts file,) or if the JRE cannot reach the address because of interference from a firewall or proxy server.
Note that a so-called "wildcard" certificate can be purchased to cover a number of servers. For example, if the CN is "*.hosted.xmatters.com", then this certificate will apply to any server whose name ends in ".hosted.xmatters.com". Be aware that the wildcard part of the name cannot contain a "." (ie, the above certificate will work with "a.hosted.xmatters.com", but not "a.b.hosted.xmatters.com".)
If the "Valid Until" date has expired, the connection will not succeed. Similarly, if there is a "
Revocation Time:", the certificate will not be accepted. Certificate issuers occasionally revoke certificates if there is a problem with them - for example, if a vulnerability such as heartbleed is discovered. In such cases, the server administrator should be able to get a replacement certificate from the issuer.
4.5 Testing the Connection
4.5.1 Using Ssltest
If the certificate in the trust store is valid, and the integration is configured to use the correct trust store, then the connection should work. If it appears not to be working, try ssltest will let you test the connection using the IA's JRE, but having to use the IA. This is desirable when the IA is being used in production, and also lets you avoid the clutter of the IA's console and/or logs.
To do this:
- use ssltest.class, which is attached to this KBA:
- download ssltest.class and copy it to <IAHOME>\jre\bin (to ensure that the test uses the same JRE as the integration agent)
- open a command window in <IAHOME>\jre\bin
- invoke ssltest with the following command:
java ssltest <url>
- for example,
java ssltest https://www.xmatters.com
- if the certificate is in a different trust store, use the following syntax to specify the trust store and password:
java -Djavax.net.ssl.trustStore=</path/to/cacerts> -Djavax.net.ssl.trustStorePassword=<password> ssltest <url>
- for example,
java -Djavax.net.ssl.trustStore=..\lib\security\cacerts -Djavax.net.ssl.trustStorePassword=changeit ssltest https://www.xmatters.com
- The expected output is:
C:\xMatters\integrationagent-18.104.22.168\jre\bin>java ssltest https://www.xmatters.com
[INFO] Received host address https://www.xmatters.com
[INFO] Setting connection timeout to 5 second(s).
[INFO] Trying to connect to https://www.xmatters.com
[INFO] Great! It worked.
- If the certificate has expired, the result will be
[INFO] Could not connect to the host address https://expired.badssl.com
[INFO] The error is: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed.
- If the certificate specifies a key that is not secure enough to comply with TLSv1.2, the result will be
[INFO] Could not connect to the host address https://dh512.badssl.com
[INFO] The error is: DHPublicKey does not comply to algorithm constraints
[INFO] Here are the details:
[SEVERE] DHPublicKey does not comply to algorithm constraints.
There is a variety of bad certificate types accessible at bad-ssl.com. By testing your troubleshooting tool against these certificates, you can reproduce the error that you are experiencing and thereby identify the cause of the error you are diagnosing.
4.5.2 Using Analyze-ssl.pl
This is another excellent tool for SSL connection diagnosis. Unfortunately, it has a couple of disadvantages:
- it requires a perl interpreter, which does not come pre-installed on most servers. Some clients might balk at installing additional software on their IA server.
- it isn't compatible with the older version of perl that installs by default on CentOS (presently 5.16.3)
Analyze-ssl.pl is compatible with the current version of ActiveState perl (5.24.1), which is available for Windows, OS X and Linux, though. You can download ActiveState perl from https://www.perl.org/get.html
If you need to diagnose an SSL connection and have access to a suitable version of perl, then this tool is well worth the trouble.
You can download the script from https://github.com/noxxi/p5-ssl-tools/blob/master/analyze-ssl.pl or you can use the copy that is attached to this document.
Syntax: perl analyze-ssl.pl <with no arguments> - display the help screeen
perl analyze-ssl.pl (options) <address> -analyze the SSL/TLS connection to the address
Example: perl d:\tools\analyze-ssl.pl --show-chain xmsupport.hosted.xmatters.com
-- xmsupport.hosted.xmatters.com port 443
* maximum SSL version : TLSv1_2 (SSLv23)
* supported SSL versions with handshake used and preferred cipher(s):
* handshake protocols ciphers
* SSLv23 TLSv1_2 ECDHE-RSA-AES128-GCM-SHA256
* TLSv1_2 TLSv1_2 ECDHE-RSA-AES128-GCM-SHA256
* TLSv1_1 TLSv1_1 ECDHE-RSA-AES128-SHA
* TLSv1 TLSv1 ECDHE-RSA-AES128-SHA
* cipher order by : server
* SNI supported : ok
* certificate verified : ok
* chain on 22.214.171.124
* [0/0] bits=2048, ocsp_uri=http://ocsp.digicert.com, /C=US/ST=CA/L=San Ramon/O=xMatters, inc./CN=*.hosted.xmatters.com SAN=DNS:*.hosted.xmatters.com,DNS:hosted.xmatters.com
* [1/1] bits=2048, ocsp_uri=http://ocsp.digicert.com, /C=US/O=DigiCert Inc/CN=DigiCert SHA2 Secure Server CA
* [-/2] bits=2048, ocsp_uri=, /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
* OCSP stapling : no stapled response
* OCSP status : good
This option, which analyzes the certificate chain, is one of the most useful features of the analyze-ssl tool (particularly because this functionality doesn't seem available at all in the Java world.) There are other options, plus a verbose mode. See the help mode for details, by invoking analyze-ssl.pl with no arguments.
If the client won't let you install this tool on the IA server, it works just as well from a laptop. Ideally, you will use a wired connection to the same switch as the IA is using, so that you won't get false results due to proxy servers, etc.
Keep in mind that this tool uses a different certificate store and different algorithms from the IA's JRE. So, as with a browser, there is a chance that you will get different results from this tool than you get from the IA or ssltest.
4.5.3 Using ssllabs.com
If the server is internet-accessible, then you can paste its URL into www.ssllabs.com to get a comprehensive report on its SSL certificate. Despite the excellent information available from this site, there are a couple of deficiencies:
- It only works with publicly-accessible servers, so if you are trying to diagnose a certificate belonging to a server on a LAN or a proxy server, then ssllabs will probably not be able to help you.
- As with alalyze-ssl.pl and the browser, ssllabs.com uses a different certificate store and different algorithms from the IA's JRE. So there is a chance that you will get different results from this tool than you get from the ssltest or from the IA itself.
5 Advanced Diagnostic Procedures
If the above tools fail to help you resolve the problem, then there are still a few things you can do.
5.1 Re-checking the Basics:
- ensure that the certificate is valid (and hasn't expired)
- ensure that you are testing with a URL that matches the Certificate Name shown in the trust store
- ensure that the integration agent server can resolve the management server's network name, exactly as it appears in the Certificate Name
- ensure that there is no interference from a firewall or a proxy server between the integration agent server and the management system
5.2 Using Ssltest in Debug Mode
If you pass the debug=all flag to the JRE while invoking ssltool, the output will be verbose but potentially very helpful in diagnosing the connection failure. By comparing the tool's debug-level output with the attached file, you will be able to determine exactly which step failed in the SSL exchange.
Open the attached ssltest.debug.log file with a text editor to see examples of the following information from a successful exchange:
- truststore name and path
- list of certificates found in the truststore (abbreviated in this example)
- SSL handshake (initial "write" from client and "read" response from server "vic-vw-remedy8.invoqsystems.com")
- analysis of the certificate chain sent by the server
- "Found trusted certificate" message indicating successful match of the server certificate with one found in the truststore
- request for data, using the secure connection
- decrypted information received from the server
- "close" message sent by client
- client closes the socket connection
Compare the information in this file with the output from your own session.
Use the following syntax to run ssltest in debug mode:
<IAHOME>\jre\bin\java.exe -Djavax.net.debug=all ssltest <URL>
To simplify inspection of the output, redirect it to a text file. For example:
<IAHOME>\jre\bin\java.exe -Djavax.net.debug=all ssltest https://www.xmatters.com/ > %temp%\ssltest.txt
Hint: when opening the output file in a text editor, search for "***". This string is used as a marker to indicate milestones in the establishment of an SSL connection.
Here are some notes on interpreting the verbose output:
- The messages do not discriminate between TLS v. 1.0, TLS v. 1.1 and TLS v. 1.2, so don't spend time investigating the possibility that TLS 1.0 is being used instead of TLS 1.2. Assuming that you are using IA 5.1.8 (which ships with JRE 8) or later, TLS 1.2 is used by default.
- Don't be deceived by the messages saying "adding as trusted cert:". Those messages indicate that the JRE is reading its trust store file and making a list of candidate certificates to be potentially matched against the ones submitted by the server. The messages don't indicate that a server's certificate will automatically be trusted just because it has the same "CN=" name.
- The "*** Certificate chain" messages are worth paying attention to. Typically the certificate chain will be an array of one or more certificates which have been submitted by the server, and the client will attempt to match those certificates to the ones in its trust store. If the chain has multiple certificates, then the first one (labeled "chain ") will be the server's own (eg, "CN=*.xmatters.com") and the rest ("chain ", etc) will be a "root" from the certificate authority (the vendor) who provided the certificates.
- The "certificate chain" messages should be followed by "***
Found trusted certificate:"
- If the "certificate chain" messages are followed instead by messages like these, then there is a problem with the certificates sent by the server:
"%% Invalidated: [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA]"
"TLSv1 ALERT: fatal, description = certificate_unknown "
The problem is likely to be one of:
- The "root" certificate from the server is not recognized as a Certificate Authority. The root cert should include a record saying "CA:true", and should match one of the certs in the IA's trust store.
- The IA's JRE could not create a valid link from the server certificate to the root certificate. For example, if the certificate chain is intended to provide an intermediate certificate as well as a root and a server certificate (which is often the case,) but the intermediate certificate is absent, then the X509 Trust Manager will not be able to validate the server certificate and the connection will fail. In other words, the certificate was not properly installed on the server. A browser should be able to confirm this diagnosis.
- The server's certificate was not added correctly to the IA's trust store. One likely cause is that the certificate was not in a form that the IA's JRE could accept (using keytool - list should tell you if this is the case.)
Another possibility is that the server's entire certificate chain was installed in the IA's trust store. It's a mistake to install the server's intermediate and root certificate in the IA's trust store. If you install the server's certificate alone (whether or not it is self-signed,) the IA will trust it - assuming that the certificate is in a format that is acceptable to the IA's JRE. Again, keystore -list will tell you this.
If these notes don't help you to resolve the problem, then take a look at the attached Oracle "Debugging SSL/TLS Connections" document for further information about the certificate validation and handshaking process.
5.3 Compiling Ssltest on the Server
If the client will not agree to installing the ssltest binary file on the server, they may agree to allowing the ssltest tool to be compiled from the source code attached to this KBA. Provide the file ssltest.java to the client - it is a very simple piece of code that can be inspected with a text editor. Once it has been inspected, the client can compile it with the following command (assuming that the Java JDK is installed):
This will create a new copy of the ssltest.class file which can be used as documented above.
6. Background Information
All these terms can be so confusing! Troubleshooting is hard enough without a bunch of jargon getting in the way. Here are explanations of a few of the terms used in this KBA.
SSL ("secure sockets layer") is a mechanism that allows HTTP requests and responses to be transmitted securely. HTTP plus SSL is where HTTPS comes from. SSL is also used in other forms of communication, such as SSH (which allows people to log on to and control remote servers,) and SCP (which allows files to be transferred securely.)
Two major parts of SSL communications are encryption and verification. Encryption ensures that a third party who intercepts a request or response will not be able to understand the content. Verification ensures that a client can trust the response that they receive from a server, by verifying the responder's identity. That is what SSL certificates are for.
TLS is a newer version of the Secure Sockets Layer specification. In this KBA I've avoided referring to TLS because the term "SSL" is woven all through the terminology (e.g., no-one uses the term "TLS certificates"), and it wouldn't have added clarity.
A keystore is just a file or other storage medium that contains SSL keys and/or certificates. The keys and certificates have to be added via software such as the Java keytool.
Wait a minute... so far we've been talking about putting "certificates", not "keys", into keystores. So what's this "key" business? Answer: an SSL key (actually, a key pair) is a unique two-part digital document that is required to create an SSL certificate (either commercial or self-signed.) The SSL key pair consists of a public key, which is included in the certificate, and a private key, which is kept on the server and is never shared.
For you keeners, there is a good overview of SSL/TLS encryption here: https://www.digicert.com/ssl.htm
A server's keystore holds its private key and its SSL certificate. When the client receives a response from the server, it uses the certificate information in the response to verify the server's authenticity. The client also uses the public key from the certificate to encrypt its next request. The server uses its private key to decrypt the request.
A trust store is also a file that contains certificates. The main difference between a keystore and a trust store is that a trust store is used by a client (i.e., the device that initiates a request to a server), whereas a keystore is used by a server. Another difference is that a keystore contains the server's public key and its certificate, whereas a trust store contains only certificates. When the client receives an HTTPS response, it compares the certificate information provided by the server with the certificates in its trust store, to see if there is a match. If none of the certificates match, then the response will not be trusted.
Different devices and systems store their trusted certificates in different ways. The Windows OS stores its trusted certificates in the Registry. Some browsers such as Chrome and Internet Explorer use the trusted certificates provided by the OS. Other browsers such as Firefox and Opera keep their own separate trust store, in a file. And the Java JRE also keeps its trusted certificates in a file, as described above.
Since the JRE can operate as a server or a client, it needs a keystore and also a trust store. By default, the JRE's keystore and trust store are the same file.
In the JRE's default trust store ("cacerts"), you will see a number of certificates that were shipped with the JRE. These certificates are in the trust store because they are issued by Certificate Authorities; generally, these are the companies that sell SSL certificates.
If a server sends a certificate that was supplied by a commercial signing authority, but the certificate was not installed correctly on the management system, then it may not be recognized by the JRE. In other words, it will be treated like a self-signed certificate. And, like a self-signed certificate, the IA's JRE will trust an improperly-installed server certificate, if it is added to the IA's trust store.
7. SSL Inspection Proxy Servers
You may encounter difficulties connecting to xMatters On Demand through an SSL inspection proxy server. A likely cause is that the proxy server removes the certificate to inspect the traffic and then puts a new certificate on the data to deliver it to the Integration Agent. The agent will not accept the data if the proxy server's certificate is not trusted (eg, if the certificate is self-signed.)
One solution is to switch off SSL inspection on the proxy for that traffic. If that's not possible, then adding the proxy server's certificate to the IA's trust store (as described in section 2) solves the problem.
(Thanks to Adam Watson for contributing this field-tested information.)
8. Further Reading
This is an excellent resource for SSL/TLS troubleshooting: https://maulwuff.de/research/ssl-debugging.html
JDN-4346 Originally created by Jeremy Brown