What's the problem?
The POODLE vulnerability is a security issue within the SSL 3.0 protocol, identified by Google in October of 2014. The researchers found that an attacker with control over network connections (such as on a public WiFi network) could trick a web browser into leaking personal cookies. In turn, these cookies could be used to assume another identity on secure web services.
The good news is that the SSL 3.0 protocol has been superseded by more secure communication protocols for at least a decade. The bad news is that the same vulnerability exists on some implementations of the Transport Layer Security protocol.
This means that xMatters will require all customers to update their local deployments to ensure that they are not attempting to communicate using SSL v3.0 or TLS v1.0.
What are we doing about it?
xMatters has determined that the best way to ensure that our users are protected from this vulnerability is to do the following:
- First, in accordance with network security recommendations, we will be disabling SSL 3.0 support on all of our servers. This effectively prevents older and out-of-date browsers from inadvertently falling prey to the vulnerability by preventing them from accessing xMatters using SSL 3.0.
- Second, we must ensure that the only applications (such as xMatters integration agents and EPIC clients) communicating with xMatters are upgraded to use protocols other than SSL 3.0. (Previous versions of the integration agent had SSL 3.0 support enabled as a back-up protocol in the event that TLS was not available, which means they may be subject to this vulnerability.)
How does it affect you?
This will affect you in one or both of the following ways:
- If you have a local xMatters integration agent in your deployment, you will need to ensure that it has been upgraded to a minimum of version 5.1.3.
- AIX integration agent deployments require a minimum version of 5.1.4.
- If you are using an EPIC client to sync your data, you must ensure it is using a minimum JRE version of 1.7.0_45.
WARNING: the latest version of the integration agent (5.1.5) has a known issue that affects some integrations. Prior to upgrading, read this knowledge base article for more information.
How long do you have to do this?
Due to the critical nature of this vulnerability, xMatters will be disabling these older versions by region, according to the following schedule:
|Australia, Japan, Asia Pacific||Monday, May 25, 2015|
|Europe, Middle East, Africa||Monday, June 1, 2015|
|North America||Monday, June 8, 2015|
On these dates, a new protocol suite will be implemented in all xMatters On-Demand deployments for each region. If you have not updated your environments at that time, you may not be able to communicate with xMatters.
How can you get more help?
If you have any questions, or need help with the upgrade, contact xMatters Support using the Submit a Request form.
How do you check your integration agent version?
On the machine hosting the integration agent, open a command window and navigate to the <IAHOME>/bin folder. Then, execute the following command:
./iadmin.sh get status
If the integration agent is running, you should see an output that resembles the following:
Version: 5.1.3-SNAPSHOT r79677 Release date: 07-Jan-2015 Agent started: 10-Apr-2015 13:07:12 Agent ID: vic-vm-mbennett/10.2.1.168:8081 Integration Services: Event domain: applications Name: sample-relevance-engine Clients: [MG, APCLIENT] URL: http://10.2.1.168:8081/applications_sample-relevance-engine Started: 10-Apr-2015 13:07:13 Last request: none Status: ACTIVE Pending request count: 0 Normal priority inbound APXML queue size: 0 High priority inbound APXML queue size: 0 Normal priority outbound APXML queue size: 0 High priority outbound APXML queue size: 0 Event domain: default Name: sample Clients: [MG, APCLIENT] URL: http://10.2.1.168:8081/default_sample Started: 10-Apr-2015 13:07:13 Last request: none Status: ACTIVE Pending request count: 0 Normal priority inbound APXML queue size: 0 High priority inbound APXML queue size: 0 Normal priority outbound APXML queue size: 0 High priority outbound APXML queue size: 0 Event domain: del Name: del Clients: [APCLIENT] URL: http://10.2.1.168:8081/del_del Started: 10-Apr-2015 13:07:14 Last request: none Status: ACTIVE Pending request count: 0 Normal priority inbound APXML queue size: 0 High priority inbound APXML queue size: 0 Normal priority outbound APXML queue size: 0 High priority outbound APXML queue size: 0 Event domain: generic Name: generic Clients: [MG, APCLIENT] URL: http://10.2.1.168:8081/generic_generic Started: 10-Apr-2015 13:07:14 Last request: none Status: ACTIVE Pending request count: 0 Normal priority inbound APXML queue size: 0 High priority inbound APXML queue size: 0 Normal priority outbound APXML queue size: 0 High priority outbound APXML queue size: 0 Event domain: ping Name: ping Clients: [MG, APCLIENT] URL: http://10.2.1.168:8081/ping_ping Started: 10-Apr-2015 13:07:13 Last request: none Status: ACTIVE Pending request count: 0 Normal priority inbound APXML queue size: 0 High priority inbound APXML queue size: 0 Normal priority outbound APXML queue size: 0 High priority outbound APXML queue size: 0 xMatters Servers: URL: http://10.2.1.36:8888/api/services/AlarmPointWebService Connectivity status: PRIMARY_ACCEPTED Last heartbeat attempt: 10-Apr-2015 13:52:58 URL: http://10.2.1.36:8888/api/services/AlarmPointWebService Connectivity status: UNKNOWN Last heartbeat attempt: none
The version of the integration agent is reported in the first line of the output; it must be "VERSION: 5.1.3-" or higher.
How do you verify that your integration agent is working properly?
First, you can use the output you generated to check the health of the integration agent; if the integration agent is properly connected, you should see PRIMARY_ACCEPTED on at least one of the defined xMatters servers.
Second, check the <IAHOME>/log/AlarmPoint.txt file and ensure it does not contain any errors.
Third, inspect the log file for activity: if the integration agent is active and in use, you should see log entries for events coming in and appearing in xMatters.
If there is no activity or the integration agent isn't currently in use, you can run a test injection using the out-of-box ping integration (if the ping endpoint has not been disabled):
- In the <IAHOME>/bin folder, run the following command (replacing <xm_user> with the user ID of a user with at least one active device in xMatters, and <server_ip> with the IP address of a server that can be pinged from the integration agent machine):
APClient.bin --map-data ping <xm_user> Test <server_ip> INCOMTEST0001
- Check the <IAHOME>/log/AlarmPoint.txt file to ensure there are no errors, and that the integration agent created the event.
- Confirm that the event is created in xMatters, and the user is notified.
- Respond to the notification on the device, and confirm that the integration agent processes the response (it should return the ping results to the user).
Note: Make sure you repeat these steps for all integration agents in your deployment. If the ping integration endpoint has been disabled on your system, replace the above command line with an appropriate one for your integration.
How do you upgrade your integration agent?
Start by downloading the latest version from the integration agent product page.
Complete instructions for upgrading your integration agent are contained within the release notes for each version:
The basic steps are:
- Back up your existing files, including any integrations or other customizations.
- Extract the integration agent download package to create a new integration agent installation folder.
- Copy the necessary configuration files to the new integration agent folder.
- Merge any integrations and customizations.
Some older integrations relied on a specific version of the IAUtils package that was bundled with versions of the integration agent prior to 5.1.3. If you are upgrading an integration agent version 5.1.2 or lower, and have an existing integration, you may need to restore your IAUtils file. For more information, refer to Integration agent utilities.
How do you make sure you are protected?
First, stop your existing integration agent if you haven't already, and start the new version. Then check the <IAHOME>/log/AlarmPoint.txt file, and ensure there are no errors.
Next, run the get-status command as explained above and check the version number (must be 5.1.3 or later!) and ensure at least one of the xMatters servers identified at the bottom of the output contains PRIMARY_ACCEPTED. You can also follow the other steps outlined in the verification process above.
What about EPIC data sync?
If you are using EPIC to synchronize user and group data, we recommend that you use the version of EPIC that matches your version of xMatters On-Demand.
You should also ensure that you are using a minimum JRE of version 1.7.0_45, which is compatible with TLS version 1.2.
Note: It is possible to have two versions of JRE installed on the same machine. Use the following instructions to determine which version your EPIC client is using.
The first step is to determine whether the JAVA_HOME environment variable is set for your EPIC client. If this is set, the EPIC client will always use the JRE set for JAVA_HOME. On the machine where your EPIC client is installed, run one of the following commands:
If it returns a value, run
The version specified will be the one used by the EPIC client.
If the echo command does NOT return a value, it means that the JAVA_HOME environment variable is not set. To get the JRE version on your machine, run:
The JRE version returned by either command should resemble the following:
java version "1.7.0_75" Java(TM) SE Runtime Environment (build 1.7.0_75-b13) Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)
If the Java version identified in the first line of the output is not at LEAST 1.7.0_45, you will need to update to a minimum of Java 7; we recommend the latest update available.
xMatters internal references
DTN-4292, COR-2362, COR-2410, SUP-9967, COREL-112