Improving our hosting services

We launched xMatters On-Demand way back in 2010, which in high-tech time is roughly when the Flintstones were cohabitating with dinosaurs. Back then, many of our customers were in their own early stages of cloud adoption with requirements still best suited to xMatters data centers at colocation facilities. Seven years later, software services have changed drastically, with even the most security-focused banking, healthcare, and government agencies adopting some form of approved cloud computing.

To meet this dynamically shifting landscape, we've constantly evolved our solution to drive out complexity, provide new features, and offer ever-increasing levels of performance. Here are just a few examples of those efforts:

  • The creation of new cloud-friendly features such as the Communication Plan Builder, which allows our customers to easily design their communication processes and the integrations that drive them.
  • Architecture changes to accommodate new design patterns like a micro-services orientation to optimize performance, and to provide new features such as in-country SMS codes (introduced in September 2014 when we launched our first micro-service for SMS delivery).
  • Deployment process updates that allow us to deliver fixes and features faster and more efficiently - and then even faster and even more efficiently again.

These changes have allowed our service to grow and change in concert with our customers' shifting needs. Now, to best support current and future needs, our services are evolving again and our underlying infrastructure will be undergoing important upgrades and enhancements.

Why upgrade?

In short, we're upgrading our private infrastructure as a service provider to make sure we can support all of our customers, no matter their size or requirements. Consider the following:

  • Scale - We have hundreds of customers, each with their own unique needs for toolchain integration, data synchronization, notification, and other resource-intensive operations. 
  • Elasticity - Need for scale is unpredictable and may require our services to dynamically expand and contract as our customers deal with event floods from monitoring systems, updates from the HR system, large notification requests for major incidents, and similar usage spikes. 
  • Availability - Communications are a critical component of IT process and cannot afford downtime. 
  • Analytics - Vast amounts of performance and behavioral data pour into and through the xMatters system, allowing our customers to gain important insights into their processes.

And, of course, the biggest point of all:

  • Security - Security is a concern for everybody, in the cloud or otherwise. We're building out an ISO 27001-compliant system, and supporting multiple hosting regions to help meet data localization requirements. We'll be able to offer new or improved versions of our data-at-rest cryptographic and physical protection, secure browser communications, and privacy and security program.

What's the process?

The process of migrating to our new services will be similar to our existing processes for failover. In advance of the migration, services and data will be created and replicated as necessary. The old and new environments will be kept up to date through our continuous delivery processes.

During migration, xMatters services will be unavailable for a period of time. Because the process is similar to our existing data center failover, customers can expect a similar duration for the operation. xMatters is currently testing and refining the migration process and more detailed information will be available based on our testing.

Non-production environments will be migrated in advance of production migrations. The non-production migrations will allow for customers to perform any testing or validation required in advance of the production migration.

For more information about the specific process and communications, see the maintenance notice here.

When is this happening?

The timing of each migration will be dictated by your hosting region with the following date windows:

Region

Non-Production

Production

Europe, Middle East and Africa

COMPLETED!

May 8-11, 2018

 

May 22-25, 2018

 

Asia Pacific

COMPLETED

October 16-18, 2018

October 23-25, 2018

Americas

COMPLETED

December 11-14, 2018*

December 17-21, 2018**
January 7-11, 2019
January 14-18, 2019

  * Except customers with legacy integrations or certain requirements.
** Applies to a specific set of migration-ready customers only.

Note: Some customers with unique requirements or concerns will be upgraded on their originally scheduled dates.

How do you get ready?

We understand that change can be a pain - we're big consumers of evolving technologies ourselves. The good news for migration is that there's a lot of long-term gain for a bit of up-front pain. If you'll allow us to get a bit geeky for a moment, consider that Lehman's laws of software evolution are just as valid today as they were when they were first written decades ago.

And, in our case Lehman's Law of Increasing Complexity applies to this improvement: as a system evolves, its complexity increases unless work is done to maintain or reduce it. Migrating our service allows us to replace older components with modern technologies - and we believe now's the time to reduce the complexity of our system while offering customers the latest capabilities.

The following sections outline the changes that will require customer action.

Networking and Email

Any optimizations you have created in your own networking, email routing or other systems that use hardcoded addressing techniques will need to be evaluated and potentially modified to point to the new xMatters environments.

IP Filtering

xMatters strongly discourages the use of IP filters. Our experience with providing emergency communications for incidents of all types, including disaster recovery and business continuity situations, has taught us that any mechanism that restricts the immediate and critical flow of communications can be a hindrance - especially in a crisis. Although IP filtering is often seen as part of a "Defense-in-Depth" strategy for traffic shaping, the dynamic services offered by cloud providers - including xMatters - may not perform optimally with these strategies. In fact, the high-availability protocols used by cloud services, such geographic and high-IP-range load balancing, can produce a denial-of-service (DOS) issue if the load balancing systems require an IP change.

To help provide the best security possible without impacting the free flow of events and notifications, we provide a full range of application-level access management and control, including native and federated secure login, and data-in-transit protection via HTTPS/TLS. 

Customers with stringent security policies that are using IP lists or other filtering will require customer side changes in advance of the migration to ensure they don't experience a service interruption. We will work toward a solution for customers who absolutely must have specific IP ranges, however we cannot identify when or if our customers are using this type of customization. You will need to proactively identify if you need to modify your network settings.

For more information, see our article about IP filtering.

SSL updates

These improvements include some enhancements to our SSL infrastructure to keep up with current security trends and requirements. As part of these changes, we're removing some outdated SSL ciphers from the list of allowed ciphers.

You will need to make sure you are not using the weaker ciphers; if you have not updated your configuration before the hosting service improvements are complete in your region, your connections will be refused.

For more information, see our SSL changes and updates article.

End of Life for Advanced Messaging (legacy scenarios)

Advanced Messaging is - or, rather, was - a module for saving and recalling message content, recipients, and responses so communication processes could be predefined for important incident response activities. The feature was a precursor to our Communication Plan Builder and lacks many of the abilities that our newer communication plan forms and their scenarios support, including:

  • Purpose-built forms with input validation to gather the exact information you need to drive your incident response
  • Drag-and-drop design tools for building message templates for text (SMS), voice, email, and mobile app push content
  • Mobile app initiation and status reporting
  • Map-based recipient targeting
  • Message scheduling
  • And much more...

Customers will need to re-implement any advanced messaging content to form-based scenarios - the benefits in usability and functionality are already reason enough to make the switch! As an added benefit, this consolidation of our feature set into the newer technology means we can focus on refining enhancements, performance tuning, and bug squashing to constantly improve your xMatters experience.

For more information, check out our end-of-life announcement.

End of Life for Action Scripting

Action Scripts and the event domains they live in were our mechanism for defining business logic, message formats, responses, and integration behaviors. This proprietary language has been replaced with our drag-and-drop communication plan design tools, and the Integration Builder offers a much better method for building custom integrations. And, you'll recall that our Integration Directory offers over 60 pre-built, easily-configured integrations to common IT systems - with more being added all the time.

Customers will need to re-implement any integration that uses Action Script to either an integration from our Integration Directory, or an integration built using our communication plan design tools. Consolidating your event traffic to flow only through our newer technologies allows us to provide an event pipeline with enhanced capabilities for filtering, suppression, correlation, and prioritization so that each communication process can reliably find the right recipients.

For more information, check out our end-of-life announcement.

Changes to Customer Hostnames and URLs

We're simplifying the URL used to access xMatters by removing extraneous hosting information. In addition to improved usability, this change allows us to make important improvements to our infrastructure, including enhancements to performance, security, reliability, and data retention.

For most of our customers, there's nothing you need to worry about right away - though we do encourage you to start using your new and improved hostname. For a few of our customers, this change might cause a conflict if they have similar hostnames for different locations, such as "example.hosted.xmatters.com" and "example.na1.xmatters.com". 

We have already identified the clients that will be impacted by this change, and we'll be working closely with them to select a new hostname, and to devise a project plan specific to their individual inventory of affected components.

See a complete explanation of this change in our announcement here.

Call-in numbers

We're also taking this opportunity to enhance our telephone access points to replace some older technology. This means that we'll need to change a few phone numbers at the same time, including some of the numbers that people call in retrieve their voice notifications.

We've already reached out to each customer affected by this change and provided the new numbers they need to call and guidance for configuring their systems. The new numbers are already operational, and displayed as the caller ID for anyone receiving voice notifications from xMatters.

Spam Filters

Our new service also includes hypersensitive detection for email bounces, which enhances our ability to identify and disable invalid email devices. This feature uses the "Return-Path" header to indicate where the emails should bounce, but the header value being sent can often be longer than some spam filters like to see - especially when they're configured with very strict settings. This could potentially cause some email from xMatters to be incorrectly flagged as spam. 

We suggest that our customers evaluate their spam filter settings to ensure that they won't inadvertently flag xMatters email due to the long "Return-Path" values.

Delayed DNS Propagation for users in China

Our experience has taught us that the time-to-live (TTL) of DNS entries is not always respected for users in China. For example, even though our TTL is set to five minutes, some users on Chinese carriers have not seen DNS changes propagated for up to four days. 

When we begin the migration process for the APJ region, this discrepancy could cause some users' computers to look for xMatters at the old hosting locations, rather than the new ones. Those users would encounter an error message, and would not be able to connect to or use xMatters. This problem will resolve itself over time, but we will be actively monitoring the situation, and reaching out directly to customers we believe may be experiencing this issue.

Other Changes

In addition to these changes we strongly recommend the following when building new integrations:

  • Integrate events using our Integration Builder. The REST API's POST trigger end point will eventually be deprecated with the expectation that all integration traffic will use the Integration Builder. We've made this very simple with the addition of our "Create a new xMatters event" action in inbound integrations - it understands exactly the same JSON payload as the REST call.
  • Use xM API calls for web services work. Our SOAP API and older REST API will eventually be deprecated. While we haven't yet provided deprecation dates, using these older technologies will mean more work later on to move to the equivalent xM API calls.

How do you get help?

We are ready, willing, and eager to help you however we can!

We've already engaged with a number of customers that are preparing for the migration and we're finding that they frequently want to re-evaluate their integration points to best support their IT processes and to use our latest integrations.

If you already know what you'd like to do but need technical assistance with our supported integrations or changes to your deployment, contact xMatters Support by submitting a request.

Was this article helpful?
0 out of 0 found this helpful

Comments

1 comment

Article is closed for comments.

  • Added additional information around updates to spam filters, and a warning about potential DNS issues for some users in China.

    0